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ABSTRACT 


Horn,  Jay  Ashley.  M.S.,  Department  of  Biomedical  and  Human  Factors 
Engineering,  Wright  State  University,  1989.  A  Theory-Based  Methodology  for 
Analyzing  Domain  Suitability  for  Expert  Systems  Technology  Applications. 

This  thesis  chronicles  the  development  of  a  theory-based  methodology  for 
analyzing  candidate  application  domains  for  expert  systems  technology  solution. 
There  is  clearly  a  need  for  a  methodology  of  selecting  expert  systems  application 
domains  that  has  firm  theoretical  underpinnings.  To  support  that  goal,  four  theories 
of  human  cognition  are  evaluated.  Gestalt  theory,  stimulus-response  theory, 
information  processing  theory,  and  the  Structure  of  Intellect  theory  (SOI)  are 
assessed  against  the  criteria  of  construct  validity,  reliability,  and  operational  utility. 

The  SOI  theory  is  selected  as  the  framework  for  identifying  the  kinds  of 
information  and  mental  processes  (the  information  elements)  essential  to 
satisfactory  performance  in  a  given  domain.  The  SOI  theory  forms  the  basis  of  the 
Domain  Suitability  Analysis  Tool  (DSAT).  By  using  an  established  theory  of  human 
cognition,  the  methodology  enjoys  a  high  level  of  construct  validity  for  describing 
the  knowledge  components  of  a  domain. 

The  DSAT  is  hypothesized  to  specify  the  information  requirements  of  a  domain, 
present  the  data  in  an  easily  interpreted  format,  and  possess  both  reliability  and 
validity.  The  DSAT  outputs  can  then  be  used  to  assess  the  degree  of  supportability 
for  the  domain  requirements  based  upon  the  current  level  of  expert  systems 
technology.  This  allows  the  decision-maker  to  assess  the  degree  of  suitability  of  this 
domain  for  an  expert  systems  technology  solution. 

The  DSAT’s  operational  utility  and  reliability  are  investigated  in  two  separate 
studies.  Results  of  the  studies  indicated  a  significant  degree  of  operational  utility 
and  reliability  among  domain  experts.  Additionally,  findings  of  the  reliability  study 

iii 


offered  some  degree  concurrent  validity  with  research  being  conducted  in  the  field 
of  cognitive  science. 

A  discussion  of  the  potential  contribution  of  the  DSAT  methodology  in  such 
areas  as  knowledge  acquisition,  cognitive  engineering,  and  functional  allocation  of 
tasks  between  man  and  machine  is  presented.  Future  research  issues,  such  as 
refinement  and  automation  of  the  tool,  additional  validation,  and  empirical 
development  of  the  Domain  Suitability  Index  factor  weights  is  also  discussed. 
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1.0  INTRODUCTION 


The  science  of  artificial  intelligence  and  its  related  subfields  of  expert  systems, 
robotics  and  machine  vision,  and  natural  language  processing  have  recently  exploded 
into  the  forefront  of  scientific  and  engineering  research  and  applications.  Expert 
systems,  defined  as  knowledge-based  systems  that  emulate  expert  performance  to  solve 
significant  problems  in  a  particular  domain  of  expertise  (Sell,  1986),  are  currently 
receiving  the  a  great  deal  of  attention.  Since  many  excellent  treatments  exploring  the 
characteristics  of  expert  systems  have  been  written  (Waterman,  1986  and  Sell,  1986)  no 
formal  examination  of  expert  systems  will  be  offered.  However,  the  basic  process  of 
designing  expert  systems  will  be  examined,  together  with  the  implications  of  that  design 
process  relative  to  selection  of  application  domains  suited  to  expert  system  techniques. 

In  the  pages  that  follow,  competing  theories  of  human  cognition  will  be  evaluated 
against  the  criteria  of  validity,  reliability,  and  operational  utility  to  select  a  theory 
capable  of  defining  the  information  requirements  of  specific  domains.  After  specifying 
a  taxonomy  of  the  domain  information  elements  that  directly  contribute  to  domain 
performance,  a  methodology  for  comparing  the  degree  of  support  offered  by  expert 
systems  technology  will  be  developed. 

The  procedures  necessary  to  perform  an  accurate  evaluation  of  potential 
application  domains  for  expert  systems  technology  using  the  theory-based  model  have 
been  integrated  to  create  the  Domain  Suitability  Analysis  Tool  (DSAT).  The  DSAT 
has  been  designed  to  be  both  theoretically  valid  and  operationally  efficient.  Validation 
of  the  DSAT  is  achieved  by  examining  several  diverse  application  domains  (to  which 
expert  systems  suitability  had  previously  been  established)  and  comparing  the 
recommendations  of  DSAT  against  the  existing  knowledge  of  the  domain’s  suitability. 
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Additional  benefits  of  the  DSAT  process  are  discussed,  as  well  as  areas  for 
additional  improvement.  With  an  overview  of  the  DSAT  development  in  mind,  it  is 
appropriate  to  formally  define  the  problem  investigated  in  this  research  effort. 

1.1  PROBLEM  STATEMENT.  There  are  no  theoretically  based,  empirically 
established  methods  for  determining  domain  suitability  for  applying  rule-based  expert 
systems  technology.  While  several  ad  hoc  guidelines  have  been  proposed,  at  best,  they 
provide  only  general  guidelines  for  determining  if  a  domain  can  benefit  from  rule- 
based  implementations  of  expert  systems  technology.  Hadzikadic,  Yun,  and  Ho  (1987) 
succinctly  state  the  problem: 

As  expert  systems  technology  becomes  increasingly  popular  for  (yet)  untested 
applications,  a  serious  gap  in  knowledge  has  become  increasingly  prominent  - 
the  appropriateness  of  the  match  between  a  prospective  application  domain  and 
the  tools  of  ES  (expert  systems)  (p.  64). 

A  methodology  which  allows  the  knowledge  engineer  (or  program  manager)  to  accurately 
determine  the  degree  of  success  achievable  by  pursuing  an  expert  systems  technology 
solution  in  a  particular  problem  domain  is  the  goal  of  this  research. 

Attaining  the  above  goal  can  facilitate  the  application  of  scarce  resources  to  areas 
where  the  greatest  utility  would  be  derived.  The  tool  can  act  as  a  funnel  to  allow 
program  managers  to  direct  time,  capital,  and  personnel  into  domains  suited  to 
artificial  intelligence  techniques.  Similarly,  it  can  assist  decision-makers  in 
determining  the  level-of-effort  for  domains  showing  possible  success,  while  also 
identifying  areas  of  higher  risk. 

Another  potential  outcome  of  this  research  is  an  increased  understanding  of  the 
importance  of  the  user’s  unique  information  requirements  to  perform  the  task  and 
their  impact  on  the  design  of  rule-based  expert  systems.  Addis  (1985)  has  indicated 
that  a  more  complete  understanding  (rooted  in  theory  and  establishment  of  in-depth 
techniques)  of  the  human  factor  in  the  process  of  expert  system  design  is  required  if 
the  science  is  to  move  forward. 
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Kidd  and  Sharpe  (1988)  are  also  concerned  with  the  lack  of  theoretically  based 
research  which  accounts  for  human  expertise  in  artificial  intelligence.  They  regard  the 
current  generation  of  expert  systems  as  experiments,  which  have  focused  impressive 
amounts  of  computational  power  on  specific  problems  in  highly  isolated  domains,  but 
have  yet  to  achieve  sufficient  basis  in  theory  to  allow  growth  into  diverging  domains. 
They  write: 

Success  of  the  system  is  directly  determined  by  the  appropriate  representation 
and  application  of  specific  knowledge  from  that  domain  to  solve  an  isolated 
problem.  Despite  the  vast  amount  of  data  now  available  as  a  result  of  these 
experiments,  we  are  still  unable  to  explain  the  "why"  or  "how"  of  successful 
systems  or  to  predict  for  which  other  domains  and  tasks  the  current  techniques 
will  work.  This  is  because  no  theory  of  tasks  or  domains  currently  exists  (p. 
147). 

The  taxonomy  of  the  domain  information  structure  that  is  specified  by  this  research 
can  be  useful  in  identifying  critical  areas  of  the  domain  deserving  additional  attention 
during  the  expert  systems  design  process.  The  taxonomy  can  also  be  used  to  estimate 
the  allocation  of  functions  between  the  operator  and  the  machine,  based  upon  the 
structure  and  processing  required  of  specified  information  elements. 

The  proposed  theory-based  methodology  discussed  in  this  thesis  is  designed  to  aid 
decision-makers  in  determining  which  application  domains  hold  the  greatest  promise 
of  solution  by  rule-based  expert  system  techniques.  Additional  benefits  of  this  research 
include  focused  attention  on  the  human  element  of  the  expert  system  design  process, 
use  of  theory  to  advance  artificial  intelligence  and  expert  systems  science,  assistance  in 
identifying  man-machine  allocation  issues  (functional  allocation  of  resources),  and 
identification  of  deficient  areas  of  expert  systems  technology  requiring  additional 
research. 

1.2  TYPICAL  DEVELOPMENT  OF  AN  EXPERT  SYSTEM.  In  order  to  better 
understand  the  potential  contribution  of  this  research,  it  is  important  to  first  discuss 
how  expert  systems  are  developed  and  the  intended  impact  the  proposed  tool  will  have 
on  the  process.  The  development  of  an  expert  system  typically  follows  a  logical 
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progression  from  problem  identification  to  system  release.  Allen  (1987)  has  reviewed 
the  development  process  as  described  by  five  expert  systems  researchers.  The  five 
methodologies  are  presented  in  Table  1. 

While  using  different  terminology,  each  of  the  authors  agree  some  type  of  problem 
identification  is  required  in  the  initial  step.  The  focus  of  the  activities  performed  in 
this  first  step  one  is  not  identical  between  authors,  however.  Polit  (1985)  emphasizes 
the  "problem  recognition"  step  which  examines  both  potential  problem  domains  and 
the  appropriateness  of  expert  systems  technology  as  a  problem  solution.  Waterman 
(1986)  also  suggests  the  appropriateness  of  the  problem  domain  relative  to  expert 
systems  techniques  is  critical  in  the  system  development  process.  The  Systems 
Manufacturing  Technology  Group  recognizes  expert  systems  are  not  applicable  to  all 
disciplines  and  only  applicable  domains  should  be  pursued  (Allen,  1987). 

Hayes-Roth,  Waterman,  and  Lenat  (1983)  stress  the  identification  step  as  central  to 
the  entire  expert  system  development  project.  Here,  knowledge  engineers  evaluate  the 
structure  of  the  problem’s  tasks  and  strategies  in  terms  of  compatibility  with  the 
proposed  expert  system’s  capabilities.  Frieling,  Alexander,  Messick,  Rehfuss,  and 
Schulman  (1985),  do  not  specifically  address  domain  evaluation  relative  to  expert 
systems  as  an  important  facet  in  the  development  process. 

1.3  CURRENTLY  USED  CRITERIA  FOR  DOMAIN  SELECTION.  With  the 
understanding  that  domain  analysis  is  appropriate  and  necessary  to  the  development  of 
an  expert  system,  it  is  useful  to  examine  the  methods  currently  used  in  domain  analysis. 

The  criteria  now  in  use  derive  from  early  theoretic  models  and  guidelines. 

1.3.1  EARLY  THEORETIC  APPROACHES  OF  DOMAIN  SELECTION.  The 
early  theoretic  approaches  lay  the  ground  work  of  the  present  research.  For  example, 
the  Additive  Rating  Model  Methodology  (ARMM)  of  Bringelson,  Deer,  McCray, 

Thompson,  and  Salvendy  (1987)  attempted  to  define  a  method  of  specifying  job  tasks 
and  skills  in  terms  of  expert  systems  capabilities  (much  in  the  manner  of  a  job 


TABLE  1 

Summary  of  the  Steps  in  the  Development  of  an  Expert  System 
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Polit 
(1985) 

Step  1:  Problem  Recognition 
Step  2:  Task  Definition 
Step  3:  Initial  Design 
Step  4:  Knowledge  Acquisition 
Step  5:  Knowledge  Maintenance 


Hayes-Roth,  et  al.  and  Waterman 
(1983)  (1986) 

Identification 

Conceptualization 

Formalization 

Implementation 

Testing 


Freiling,  et  al. 

(1985) 

Phase  1:  Knowledge  Definition 
Step  1:  Familiarization 
Step  2:  Knowledge  Organization 
Step  3:  Knowledge  Representation 
Phase  2:  Prototype  Implementation 
Step  4:  Knowledge  Acquisition 

Step  5:  Strategic  Decision 
Step  6:  Interface  Design 


Systems  Manufacturing  Technology  Group 
(1982) 

Phase  1:  System  Design 

Phase  2:  Prototype  Development 

Phase  3:  Making  the  Expert 

Phase  4:  Evaluation  and  Acceptance 

Phase  5:  Use  in  Prototype  Environment 

Phase  6:  Development  of  Maintenance 
Plans 

Phase  7:  System  Release 


(Source:  Allen  (1986),  p.  50.) 
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analysis).  After  a  job  expert  selects  the  relevant  task  (or  tasks)  of  the  job  in  question 
from  the  task  list,  the  task/skill  matrix  is  used  to  evaluate  the  likelihood  of  success 
using  an  expert  system  approach. 

The  task  list  (from  Waterman,  1986)  defines  11  tasks  "involved  in  knowledge- 
based  systems."  The  24  skills  involved  in  those  tasks  are  specified  from  the  work  of 
Lenorovitz,  Phillips,  Ardrey,  and  Kloster  (1984)  that  examined  skills  used  in  human- 
computer  interactions.  Table  2  illustrates  this  matrix.  The  model  does  not  address  the 
types  of  information  used  in  task  completion  and  is  a  serious  limitation. 

There  are  several  areas  that  raise  some  concerns  of  the  validity  of  this  matrix.  First, 
neither  the  task  nor  skill  lists  possess  a  firm  theoretical  basis.  While  using  the  concept 
and  basic  emphasis  of  job  theory,  both  lists  rely  primarily  on  observations  of  the 
researchers  and  may  not  capture  the  fundamental  elements  of  the  job.  Second,  given 
the  lists  measure  the  necessary  job  elements,  the  skills  represented  are  about  1984,  and 
not  reflective  of  current  expert  systems  capabilities.  A  more  useful  tool  will  possess 
flexibility  to  measure  the  domain  against  state-of-the-art  expert  systems  capabilities. 

The  product  of  the  matrix  evaluation  is  a  numerical  rating  of  the  skills  encompassed 
in  a  task  (see  Table  2).  The  scores  can  range  from  0  (no  skills  are  replicable)  to  24  (all 
skills  are  replicable).  Higher  scores  imply  more  skills  can  be  replicated  by  the 
computer  better  defining  the  task  and  resulting  in  better  task  performance.  The  scores 
for  each  task  of  a  job  are  then  summed  and  divided  by  the  number  of  tasks  to  obtain  a 
composite  score.  Bringelson,  et  al.  (1987)  define  scores  above  17  (one  standard 
deviation  above  the  mean  task  score  of  11.73)  as  "good"  (expert  systems  are 
applicable),  scores  12  to  17  as  "marginal”  (expert  systems  may  be  applicable  with 
support  from  other  considerations,  such  as  cost),  and  scores  below  12  as  "poor"  (expert 


systems  do  not  apply). 
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TABLE  2 

Skills,  Tasks,  and  Scores  Used  in  the  Additive  Rating  Model  Methodology 

Skills  that  can  be  emulated  using  current  expert  systems  technology  (Lenorovitz, 
Phillips,  Ardrey,  and  Kloster,  1984.) 

Detect  Search  Scan  Extract  Cross-Reference 

Recognize  Categorize  Calculate  Itemize  Discriminate 

Tabulate  Estimate  Translate  Compare  Interpolate 

Formulate  Integrate  Evaluate  Select  Extrapolate 

Acknowledge/  Direct/  Ungroup/  Filter 

Respond  Inform  Segregate 

Tasks  performed  in  a  job  domain  (Waterman,  1986),  their  associated  scores  of  skill 
components,  and  qualitative  suitability  (with  appropriate  score  range)  for  expert 
systems  applications. 


GOOD 

MARGINAL 

POOR 

(17.00  -  24.00) 

(11.73  - 16.99) 

(0.00-11.72) 

Diagnosis 

-23 

Instruction  - 15 

Design  - 11 

Conflict 

resolution 

-23 

Monitoring  - 11 
&  gathering 

Information  - 1 1 

Prediction 

-21 

Creating  -  8 

Planning 

-21 

Interpretation 

-20 

Delegation 

-19 

(Source:  Bringelson,  Deer,  McCray,  Thompson,  and  Salvendy,  1987.) 
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While  the  method  makes  a  useful  contribution  in  pursuit  of  a  hierarchical 
structuring  of  tasks,  there  does  appear  to  be  some  lack  of  sensitivity  in  the  middle 
range  of  scores.  For  example,  at  most  only  1 1  tasks  are  scored.  Domains  that 
encompass  all  of  these  tasks  (or  more)  would  all  be  rated  at  11.73,  the  mean, 
suggesting  expert  systems  technology  would  not  be  warranted.  The  researchers  do 
suggest  weighting  coefficients  could  be  determined  for  different  job  domains  by  the  job 
expert,  but  again  these  weights  would  have  to  be  empirically  established  and  validated. 

A  possible  low  level  of  reliability  of  the  tool  is  evidenced  by  the  high  variability  of 
ratings  of  various  job  domains  revealed  during  the  validation  of  the  ARMM.  The 
validation  of  the  tool  involved  correlation  of  responses  of  expert  systems  practitioners 
to  applicability  of  expert  systems  technology  to  a  list  of  24  job  domains.  While  the 
experts  and  ARMM  agreed  on  the  top  three  fields  (medical  diagnosis,  computer 
system  configuration,  and  equipment  fault  diagnosis)  and  bottom  three  (key  punch 
operations,  artistry,  and  music  composition)  for  expert  systems  application,  there  was 
very  little  consensus  in  the  middle  range  fields  (examples  are  sports  coaching 
strategies,  fashion  consulting,  and  telephone  directory  assistance). 

In  effect,  the  tool  confirms  much  of  what  we  already  know  (which  is  useful). 
Unfortunately,  the  ambiguous  job  domains  are  the  most  prevalent,  and  this  is  where 
the  tool  must  be  accurate.  Since  there  is  no  expert  consensus  in  any  domain,  we  must 
have  an  objective  framework,  based  in  theory,  to  give  a  clear  picture  of  the  potential 
domain  in  light  of  the  current  technology. 

While  the  overall  concept  of  the  Additive  Rating  Model  Methodology  (ARMM)  is 
laudable,  there  are  several  problems  that  encourage  additional  research.  The 
difficulties  of  achieving  high  levels  of  validity  and  reliability,  identifying  relevant 
information  types  and  processes  used  in  the  job,  and  achieving  sensitivity  among 
complex  domains  all  point  toward  the  need  for  a  theoretically- based  approach  that  can 
ameliorate  these  deficiencies  and  provide  useful  direction  for  domain  assessment. 


TABLE 3 


List  of  “Primitives”  for  Describing 
Potential  Application  Domains 


1.  Recognize 

7.  Infer 

2.  Classify 

8.  Refine 

3.  Compare 

9.  Construct 

4.  Specify 

10.  Explain 

5.  Simulate 

11.  Execute 

6.  Analyze 

12.  Prescribe 

(Source:  Hadzikadic,  Yun,  and  Ho,  1987.) 
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Some  interesting  work  that  falls  into  the  early  theoretic  category  has  been  done  by 
Hadzikadic,  Yun,  and  Ho  (1987).  They  have  developed  12  "primitives"  to  be  used  in 
characterizing  the  components  of  an  application  domain  (see  Table  3).  The  primitives 
combine  to  define  higher  level  processes  such  as  diagnosis  and  repair. 

Based  upon  the  notion  that  humans  organize  information  into  a  mental-model  and 
then  modify  the  state  of  that  mental-model  by  using  operators,  Hadzikadic,  Yun,  and 
Ho  argue  the  operators  used  can  define  the  problem-solving  process  used.  Knowledge 
of  these  operators,  or  primitives,  therefore  characterizes  the  central  aspects  of  a 
domain.  These  "building  blocks"  can  be  combined  to  represent  any  process  particular 
to  a  domain. 

While  this  approach  has  some  merit,  neithe  the  notion  of  the  mental-model"  nor 
the  primitives  derived  are  baser1  on  any  specified  empirically  derived  theoretical 
constructs.  The  model  examtnes  the  p-ocersing  aspects  of  domains,  but  like  the 
ARMM,  encounters  difficulties  in  describing  the  types  of  information  required  for 
satisfactory  domain  task  performance.  Also  the  level  of  sensitivity  of  the  method  for 
discriminating  among  complex  domains  does  not  appear  to  be  high. 

1.3.2  GUIDELINES  FOR  DOMAIN  SELECTION.  While  each  of  the  researchers 
cited  by  Allen  (1987)  identifies  the  selection  of  a  suitable  problem  domain  as  an 
important  (and  often  critical  (Prerau,  1985))  step  in  the  development  process,  none  go 
farther  than  suggesting  a  few  broad  guidelines  for  making  this  selection.  Several  ad 
hoc  guidelines  for  domain  selection  have  been  proposed.  Table  4  lists  several  of  the 
observations  cited  by  Prerau  (1985)  and  Dreyfus  and  Dreyfus  (1986)  as  typical  of  these 
guidelines. 

The  major  objection  to  these  guidelines  is  they  are  based  simply  on  ad  hoc 
observations  of  past  successful  cases.  Dreyfus  (1979)  quotes  AI  researcher  Drew 
McDermott,  "...  AI  (and  expert  systems)  is  a  field  starving  for  a  few  carefully 
documented  failures"  (p.  46).  By  only  examining  the  "successes"  of  any  endeavor. 
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TABLE  4 

Guidelines  for  Selecting  an  Appropriate  Expert  Systems 
Application  Domain 


Dreyfus  and  Dreyfus  (1986). 

1.  No  algorithmic  solution  to  the  problem  should  exist. 

2.  The  problem  can  be  satisfactorily  solved  by  human  experts  at  such  a  high  level  that 
somewhat  inferior  performance  is  acceptable. 

3.  Non-experts  have  a  high  probability  of  making  a  poor  decision. 

4.  Poor  decisions  have  significant  impacts. 

5.  The  problem  is  stable  during  the  time  taken  to  make  a  decision. 

6.  The  knowledge  domain  must  be  relatively  static. 

7.  An  expert  must  be  available  to  provide  the  knowledge  base. 

Prerau  (1985).  (This  list  is  a  sample  of  his  52  guidelines.) 

1.  The  domain  is  characterized  by  the  use  of  expert  knowledge. 

2.  Conventional  programming  approaches  to  the  task  are  unsatisfactory. 

3.  There  are  recognized  experts  that  solve  the  problem  daily. 

4.  Experts  are  probably  better  than  amateurs  in  performing  the  task. 

5.  There  is  a  need  to  “capture”  the  expertise  for  the  future. 

6.  The  task  is  neither  too  easy  nor  too  difficult  for  an  expert. 

7.  Domain  selected  offers  the  greatest  return  for  the  projected  risk. 

8.  The  task  primarily  involves  symbolic  reasoning. 

9.  The  task  requires  the  use  of  heuristics  and  may  require  the  consideration  of  many 
alternatives  or  decisions  based  on  incomplete  or  uncertain  information. 

10.  The  task  inputs  and  outputs  are  clearly  defined  at  the  outset. 

(Source:  By  listed  author.) 


useful  information  and  "lessons  learned"  are  lost.  The  conclusions  drawn  from  this 
kind  of  analysis  are  subject  to  error  and  tend  to  be  incomplete. 

A  second  problem  with  these  guidelines  is  the  lack  of  clear  recommendations 
regarding  the  utility  of  expert  systems  technology  for  a  specific  problem.  Use  of 
general  guidelines  places  the  decision-maker  in  the  position  of  making  potentially  cost¬ 
intensive  decisions  based  on  largely  subjective  estimates  using  incomplete  criteria. 

The  current  guidelines  do  not  provide  specific  guidance  regarding  functional 
allocation  of  duties  between  the  user  and  system.  The  theoretically- based 
methodology  proposed  here  attempts  to  produce  a  model  of  domain  and  expert  system 
attributes  that  can  be  used  to  examine  the  relative  strengths  and  weaknesses  of  the 
man-machine  dyad.  This  knowledge  can  be  used  in  the  manner  of  the  Fitts’  list 
(Meister,  1985)  to  better  allocate  duties  between  the  human  and  machine  on 
cognitively  oriented  topics. 

1.4  THE  NATURE  OF  THE  PROBLEM.  Some  examples  of  traditional 
methodologies  of  building  expert  systems  have  been  described.  In  the  aggregate,  each 
of  the  traditional  methods  require  some  effort  in  initially  defining  or  identifying 
relevant  aspects  of  the  candidate  domain.  This  supports  the  assumption  that  the 
information  requirements  of  a  domain  are  the  most  salient  factors  in  attempting 
definition  and  understanding  of  the  domain.  In  practice,  however,  the  identification 
and  structuring  of  the  domain  knowledge  necessary  for  adequate  performance  is  often 
gathered  using  painstaking  and  inefficient  interviews,  known  as  knowledge  acquisition. 
Unfortunately,  the  entire  success  or  failure  of  an  expert  system  is  usually  determined  in 
the  knowledge  acquisition  phase.  Any  improvements  of  this  step  would  make  a 
significant  improvement  to  the  expert  system  development  process. 

1.4.1.  EXPERT  SYSTEMS  KNOWLEDGE  ACQUISITION.  The  knowledge 
acquisition  process  is  typically  performed  by  a  knowledge  engineer  (or  engineers)  who 
interviews  and  questions  the  domain  expert  (or  experts)  using  real  and  hypothetical 
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examples  of  cases  to  determine  the  heuristics  used  by  the  experts.  When  these  "rules 
of  thumb"  are  stated  in  an  "if-then”  format,  a  rule-based  model  of  domain  expertise  is 
created  (Klahr  and  Waterman,  1986). 

This  set  of  rules  forms  the  basis  for  the  rule-based  expert  system.  If  this  set  of  rules 
is  incomplete  or  in  error,  the  system  will  not  perform  at  intended  levels.  Revision  of 
the  rule-set  and  additional  expert  interviews  (sometimes  several)  are  required  to  bring 
the  system  up  to  acceptable  levels  of  performance. 

The  problem  of  a  usually  long  and  tedious  cycle  of  interviews  and  rule-set  revisions 
has  been  referred  to  as  the  "knowledge  acquisition  bottleneck"  (Feigenbaum,  1977,  p. 

71).  Waterman  (1986)  has  observed  the  knowledge  acquisition  step  can  extend  "...  over 
a  period  of  many  months."  New  methods  of  knowledge  acquisition  that  allow  the 
expert  to  communicate  directly  with  the  machine  to  develop  rule-sets  have  been 
developed  to  circumvent  this  problem  (see  Quinlan,  1987,  for  some  examples). 

However,  the  need  to  refine  and  correct  the  knowledge  initially  gained  is  still  evident 
in  spite  of  these  new  methods  (Klahr  and  Waterman,  1986). 

Many  reasons  have  been  offered  to  account  for  the  difficulties  encountered  in 
knowledge  acquisition.  For  example,  experts  are  believed  to  have  difficulty  articulating 
the  deeper-level  rules  they  use,  experts  may  forget  to  include  rules  they  take  for 
granted  (common  sense),  and/or  experts  may  make  incomplete  or  erroneous 
assumptions.  Generally,  the  difficulties  center  around  the  problems  inherent  in  the 
interview  process  and  getting  domain  experts  to  translate  largely  intuitive  processes 
into  verbal  representations. 

1.4.2  THE  ROOTS  OF  THE  PROBLEM.  Artificial  intelligence  and  expert  systems 
researchers  have  failed  to  notice  the  traditional  approach  used  in  building  rule-based 
expert  systems  constrains  the  domain  analysis  to  only  those  elements  supported  by  the 
current  level  of  technology.  Herein  lies  the  cause  of  the  "bottleneck"  problem.  This 
bias  towards  the  limitations  of  expert  systems  technology  can  prevent  system  designers 
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from  adequately  understanding  the  domain  and  those  information  elements  critical  to 
successful  performance  in  that  domain.  Figure  1  depicts  the  steps  involved  in  the 
traditional  approach. 

In  Step  1,  the  Expert  Systems  Capabilities  Space  is  developed  through  experience  by 
the  knowledge  engineer.  As  he  performs  the  knowledge  acquisition  process  in  Step  2. 
a  Domain  Information  Requirements  Space  is  developed  based  on  rules  elicited  from 
the  True  Domain  Information  Requirements  Space  via  the  domain  expert.  In  Step  3,  the 
Expert  Systems  Capabilities  are  mapped  onto  the  Elicited  Domain  Requirements  Space, 
apparently  covering  the  entire  domain.  However,  in  Step  4,  after  actual  use,  system 
performance  is  often  found  to  be  less  than  expected.  When  the  Expert  Systems 
Capabilities  Space  is  mapped  onto  the  True  Domain  Requirements  Space,  a  significant 
portion  of  the  domain  is  often  found  to  be  unsupported,  resulting  in  poor  system 
performance. 

An  appropriate  model  of  human  cognitive  task  execution  applied  to  the  knowledge 
acquisition  process  would  result  in  a  more  complete  and  valid  representation  of  the 
domain.  Figure  2  outlines  the  use  of  such  a  model  in  the  development  process.  Steps  1 
and  2  use  theory  to  specify  and  define  both  the  domain  elements  and  capabilities  of 
expert  systems.  In  Step  3,  mapping  the  two  together  results  in  a  more  accurate 
estimate  of  the  degree  of  support  (coverage)  that  can  be  expected  by  using  a  rule-based 
expert  systems  approach. 

Development  of  theory  through  basic  research,  while  critical  to  the  evolution  and 
establishment  of  a  science,  is  resource-intensive  and  usually  does  not  generate  profits 
"in  the  current  financial  quarter."  Therefore,  vital  basic  research  is  either  not 
performed  or  is  left  to  academia.  Hence,  many  areas  that  could  benefit  from  theory 
through  improved  methods  and  products  rely  on  empirical  data,  "rules  of  thumb,"  and 
just  plain  hard  work  and  ingenuity  on  the  part  of  dedicated  systems  designers  to  fill-in 
the  theoretical  gaps  and  advance  the  state-of-the  art.  Gaines  and  Shaw  (1988)  provide 


Expert  Systems 
Capabilities 
Space 

Derived  From 
Observation 
And 

Guidelines 


Step  1 


Elicited 
Domain  Info 
[Requirements 
Space 


Step  3. 


Step  2. 


Apparent 
Coverage  Of 
Domain 
Requirements 


True  Domain 
Information 
Requirements 
Space 


Step  4. 


K.A .•  Knowledge  Acquisition 


Figure  I.  Traditional  Approach  to  Building  Expert  Systems. 


16 


True  Domain 
Information 
Requirements 
Space 


Step  1. 


Principle- 

Based 

Domain 

Information 

Requirements 

Space 


Principle- 
Based 
Definition 
Of  Expert 
Systems 
Capabilities 


Accurate 
Estimate  Of 
Domain 
Requirements 
Coverage 
Provided 

By  Expert 
Systems 

True 

Expert 

Systems 

Capabilities 

Space 


<SteplT 

N - 


Figure  2.  Principle-Based  Approach  to  Building  Expert  Systems. 


17 


an  enlightening  examination  of  the  field  of  human  computer  interaction  (HCI)  that 
proposes  several  eras  of  evolution  for  any  scientific  endeavor.  It  is  interesting  to  note 
that  they  find  HCI  and  related  fields  entering  the  "era  of  theoretical  development," 
which  supports  the  direction  of  this  research. 

1.4.3  SUMMARY  OF  THE  PROBLEM.  In  the  rule-based  expert  system 
development  process,  domains  are  typically  evaluated  using  rule-based  expert  systems 
attributes  as  the  standard  of  comparison  and  not  theoretically  derived  models  of 
human  cognitive  task  performance.  Due  to  this  bias,  no  theory-based  models  have 
been  suggested  for  use  in  examining  domains  for  rule-based  expert  systems  suitability. 
Ad  hoc  guidelines  and  some  early  theoretic  methodologies  have  arisen  to  fill  the  void, 
but  the  difficulties  of  attaining  construct  validity,  identifying  task  relevant  information 
types  and  processes,  and  attaining  sensitivity  among  complex  domains  indicate  that  a 
more  acceptable  solution  must  be  found. 

The  next  section  explores  alternative  theories  of  human  cognition  to  be  considered 
in  development  of  a  domain  suitability  analysis  tool  (DSAT). 


2.0  THEORETIC  RATIONALE  FOR  DSAT 

This  chapter  evaluates  four  of  the  leading  theories  of  human  cognition  and 
intelligence  against  the  criteria  of  validity,  reliability,  and  operational  utility  to  derive  a 
theoretical  basis  for  the  development  of  a  domain  suitability  analysis  methodology. 

The  two  objectives  of  AI  research  are  developing  models  of  human  cognition  and 
developing  intelligent  artifacts  (Sell, 1986).  The  current  emphasis  has  certainly  been  on 
the  latter  objective,  but  not  at  the  expense  of  the  former.  Dehn  and  Schank  (1982) 
state  "...  AI  is  no  longer  concerned  with  computers  at  all  (it  is)  therefore  not  about 
anything  artificial  but  is  simply  about  intelligence"  (p.  357).  Therefore,  applying  solid 
theories  of  human  cognition  should  aid  in  the  advancement  of  artificial  intelligence 
science. 

While  no  "unified"  theories  of  cognition  currently  exist  to  solve  all  the  problems  of 
artificial  intelligence  researchers,  significant  research  has  been  performed,  resulting  in 
at  least  four  schools  of  thought  (Sternberg  and  Detterman,  1986;  Sternberg  and 
Lasaga,  1981):  psychometric,  stimulus-response,  Gestalt,  and  information-processing 
theories. 

2.1  PSYCHOMETRIC  THEORIES  OF  INTELLIGENCE.  The  psychometric 
theorists  use  individual  differences  data  (refined  through  the  use  of  factor  analysis)  to 
separate  patterns  of  reasoning  ability  from  other  abilities  and  to  examine  the  various 
reasoning  skills  exhibited.  These  various  methods  have  relied  on  the  individual- 
differences  data  for  testing  and  formulating  theories  (Guilford  and  Hoepfner,  1971). 

Spearman,  who  refined  the  factor  analysis  method,  is  regarded  as  one  of  the  first 
psychometric  theorists.  He  developed  a  two-factor  theory  of  intelligence  (specifically 
for  intelligence  testing)  which  proposed  a  single,  central  element  of  intelligence,  g 
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(Spearman,  1904).  The  second  factor  was  originally  believed  to  be  a  factor  of 
intelligence  unique  to  that  specific  test,  and  would  be  undiscernable  by  any  other  test. 

However,  as  results  of  factor  analyses  mounted,  there  appeared  to  be  other  related 
factors  that  intercorrelated  more  strongly  than  initially  predicted  by  the  theory. 

Spearman  called  variables  common  to  a  specific  group  of  tests  "group  factors."  He 
attempted  to  downplay  the  importance  of  these  factors,  emphasizing  the  importance  of 
g  instead  (Guilford  and  Hoepfner,  1971). 

Thurstone  (1938),  intrigued  by  the  existence  of  group  factors,  used  over  50  different 
intelligence  tests  to  uncover  a  number  of  intellectual  factors  in  a  large  group  of  college 
subjects.  He  proposed  a  theory  of  primary  mental  abilities  (PMA)  composed  of  seven 
factors.  These  were  verbal  comprehension,  numerical  facility,  spatial  ability, 
perceptual  speed,  rote  memory,  induction,  and  deduction. 

The  next  step  in  the  growth  of  psychometric  theories  of  intelligence  came  during 
World  War  II.  J.P.  Guilford,  as  director  of  Psychology  Research  Unit  #3  of  the 
Aviation  Psychology  Research  Program,  was  asked  to  determine  selection  criteria  for 
aircrew  personnel  in  the  intellectual  area.  Examination  of  the  reasons  students 
washed-out  of  pilot  training  revealed  eight  general  psychological  constructs: 
judgement,  foresight  and  planning,  memory,  comprehension,  visualization  of  flight 
path,  spatial  orientation,  reasoning,  and  coordination  of  information.  Guilford  and 
Lacey  (1947)  performed  an  in-depth  factor  analysis  of  these  constructs,  and 
demonstrated  approximately  25  intellectual  factors. 

After  World  War  II,  Guilford  continued  investigation  of  these  factors  of  intelligence 
under  the  aegis  of  the  Aptitudes  Research  Project  for  the  Office  of  Naval  Research. 

The  early  years  of  the  project  demonstrated  all  of  Thurstone’s  primary  mental  abilities, 
as  well  as  finding  two  aspects  of  his  spatial  factor;  arrangement  of  objects  in  space,  and 
visualizing  changes  in  objects  (Guilford,  1985).  This  brought  the  list  of  abilities  to  near 
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As  the  list  of  factors  began  to  mount,  several  similarities  and  differences  became 
apparent.  Some  factors  could  be  grouped  based  on  the  mental  processes  involved, 
such  as  cognition,  memory,  and  evaluation.  Others  could  be  segregated  based  on  the 
information  used;  symbolic,  semantic,  or  visual,  for  example.  A  third  dimension 
involved  the  form  of  the  information  used;  units,  classes,  or  relations,  for  example. 

The  resulting  cubic  figure  that  encompasses  these  dimensions  is  the  Structure-of- 
Intellect  (SOI)  model  (Guilford,  1967,  1985).  Figure  3  shows  the  SOI  model  with  its 
three  dimensions:  operations,  contents,  and  products.  This  model  illustrates  the  focus 
of  psychometric  theories  on  cataloging  and  systematically  identifying  the  components 
of  intelligence  and  cognitive  behavior. 

2.2  STIMULUS-RESPONSE  THEORIES  OF  INTELLIGENCE.  Learning  is  the 
primary  aspect  of  tr.  *  stimulus-response  (S-R)  theories  of  intelligence.  Sternberg  and 
Lasaga  (1981)  identify  five  common  themes  of  S-R  theories  (from  White,  1970,  pp  665- 
666): 

a)  The  environment  may  be  unambiguously  characterized  in  terms  of  stimuli. 

b)  Behaviour  may  be  unambiguously  characterized  in  terms  of  responses. 

c)  A  class  of  stimuli  exists  which,  applied  contingently  and  immediately 
following  a  response,  increases  or  decreases  it  in  some  measurable  fashion; 
these  stimuli  may  be  treated  as  reinforcers. 

d)  Learning  may  be  completely  characterized  in  terms  of  various  possible 
couplings  between  stimuli,  responses,  and  reinforcers. 

e)  Unless  there  is  direct  evidence  to  the  contrary,  classes  of  behaviour  may  be 
assumed  to  be  learned,  manipulable  by  the  environment,  extinguishable,  and 
trainable. 

The  primary  focus  is  on  the  role  of  past  learning  in  shaping  current  decision  making. 

Also,  Sternberg  and  Lasaga  (1981)  point  out  that  the  S-R  theory  of  intelligence  is  not 
favored  in  the  study  of  intelligence,  and  is  difficult  to  apply  to  complex  problems  even 
if  it  were  in  vogue. 

2.3  GESTALT  THEORIES  OF  INTELLIGENCE.  The  dominant  theme  in  Gestalt 
theories  of  intelligence  is  structure  of  perceptions.  Koffka  (1935),  Katona  (1940),  and 
Kohler  (1947)  have  been  the  shaping  researchers  in  this  field.  Kohler  hypothesized 
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Figure  3.  The  Structure  of  Intellect  Cube  (Guilford,  1967;  1985). 


learning  occurs  as  elements  are  organized  and  structured  in  the  mind  ("restructuring  of 
the  perceptual  field").  Also,  he  emphasized  the  role  "insight"  plays  in  problem  solving. 
Koffka  and  Katona  also  stressed  organization  of  information  as  the  dominant  feature 
of  understanding. 

For  Gestaltists,  the  goal  of  understanding  is  achieved  through  purposeful 
restructuring  of  the  problem  space  to  achieve  a  meaningful  whole.  The  solution  to  the 
problem  is  "discovered"  when  the  missing  "piece"  of  the  puzzle  is  added  to  create  a 
recognizable  and  intelligible  representation  of  the  problem. 

2.4  INFORMATION  PROCESSING  THEORIES  OF  INTELLIGENCE.  Sternberg 

and  Lasaga  (1981)  identify  information-processing  theories  as  concerned  with: 

...  the  mental  processes  individuals  use  in  reasoning,  the  strategies  into  which 
these  processes  combine,  the  representations  upon  which  the  processes  and 
strategies  act,  and  the  knowledge  base  that  is  mentally  represented. 

The  information-processing  approach  is  currently  the  most  favored  approach  for 

dealing  with  issues  of  intelligence  (Sternberg  and  Salter  1982).  After  introduction 

almost  three  decades  ago  by  Miller,  Galanter,  and  Pribram  (1960)  and  Newell,  Shaw, 

and  Simon  (1960),  it  is  still  immensely  popular,  with  the  computer  revolution  a  moving 

force  behind  this  popularity.  Sternberg’s  theory  of  information  processing  is  probably 

the  most  dominant  of  the  competing  theories.  (Sternberg  and  Lasaga,  1981,  Wolman, 

1985,  and  Sternberg  and  Detterman,  1986,  all  provide  useful  background  into  the  other 

theories). 

Sternberg  (1985)  postulates  three  aspects  of  information  processing  components, 
shown  in  Table  5.  Metacomponents  are  the  highest  level  functions  of  processing. 
Analogous  to  the  executive  control  program,  the  ten  metacomponents  direct  the 
selection  of  lower  level  performance  processes  and  assess  the  effectiveness  of  those 
selections. 

The  lower  level  of  information  processing  components  are  the  performance 
components.  These  components  execute  the  selected  metacomponent  strategies  for  a 
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TABLE  5 

Sternberg’s  Theory  of  Information  Processing 


Three  components  of  information  processing: 

I.  The  Metacomponents  are  higher-level,  executive  processes  and  consist  of: 

1.  Identifying  a  problem  exists. 

2.  Identifying  the  nature  of  the  problem. 

3.  Selecting  lower-order  components  to  solve  the  problem. 

4.  Selection  of  a  strategy  of  task  completion  using  the  lower  components. 

5.  Selecting  a  mental  representation  of  the  information. 

6.  Decision  of  allocating  attentional  resources. 

7.  Monitoring  current  position  in  achieving  the  steps  of  problem  solution  determined 
in  4,  above. 

8.  Understanding  the  quality  of  task  performance  through  internal  and  external 
feedback. 

9.  Knowing  how  to  act  upon  the  feedback  received. 

10.  Implementing  the  action  based  on  the  feedback  results. 

11.  The  Performance  Components  consist  are  lower-level  processes  used  in  the 
execution  of  task  performance  strategies  and  examples  are: 

1.  Encoding  the  nature  of  a  stimulus. 

2.  Inferring  the  relations  between  two  stimuli. 

3.  Applying  a  previously  inferred  relation  to  a  new  situation. 

III.  The  Knowledge-acquisition  Components  are  processes  used  in  learning  informa¬ 
tion  and  storing  it  in  memory  and  consists  of: 

1.  Selective  encoding,  the  selection  of  relevant  information. 

2.  Selective  combination,  which  involves  organizing  and  combining  the  new  informa¬ 
tion  to  maximize  coherence  and  connectedness. 

3.  Selective  comparison,  the  relation  of  new  information  to  information  already  in 
memory. 


(Source:  Sternberg,  1985.) 


given  task.  Examples  include  inferring  relations  between  two  events,  or  applying 
previously  inferred  relationships  to  new  scenarios. 
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The  knowledge-acquisition  components  are  those  mental  processes  used  to  store 
and  learn  new  information.  The  three  most  important  components  of  this  aspect  are 
selective  encoding  (sorting  relevant  from  irrelevant  information),  selective 
combination  (the  combination  of  selectively  encoded  data  to  result  in  highly  organized 
and  coherent  memory  store)  and  selective  comparison  (selectively  encoded  and 
combined  information  is  mapped  to  existing  memory  elements  structure  to  facilitate 


recall  and  connectedness). 


3.0  EVALUATION  OF  INTELLIGENCE  THEORIES  FOR  DSAT 

This  section  explores  the  criteria  used  in  assessing  the  appropriateness  of  the  four 
theories  of  human  intelligence  and  cognition  described  in  the  previous  chapter  for  use 
in  identifying  the  types  of  information  and  processes  of  a  domain. 

3.1  CRITERIA  FOR  EVALUATION.  The  four  approaches  examined  above  have 
different  strengths,  weaknesses,  and  intuitive  appeal.  For  use  in  DSAT,  the  theory 
selected  must  not  only  meet  the  traditional  standards  of  validity  and  reliability,  but  also 
provide  unambiguous  definition  of  the  principle  aspects  of  the  domain,  be  sensitive  to 
differences  in  the  domains,  and  lend  itself  to  ease  of  application  and  development  as  a 
tool.  Possessing  validity  and  reliability  makes  the  tool  functionally  sound.  In  addition, 
possessing  the  qualities  of  unambiguous  definition,  sensitivity,  and  ease  of  application 
makes  the  tool  operationally  practical  and  useful. 

3.1.1  VALIDITY.  The  concept  of  validity  attempts  to  define  the  degree  an 
instrument  or  model  measures  what  it  is  supposed  to  measure.  Anastasi  (1982)  reports 
the  three  principal  categories  of  this  important  measure  as  content,  criterion-related, 
and  construct  validity.  Anastasi  continues  that  validity  cannot  be  established  for  an 
instrument  in  a  vacuum;  validity  must  be  established  in  light  of  a  particular  application. 
The  relevance  of  each  category  with  respect  to  the  proposed  tool  is  examined  below. 

3. 1.1.1  CONTENT  VALIDITY.  Content  validity  seeks  to  establish  the  degree  the 
instrument  covers  the  behaviors  evidenced  in  a  particular  domain.  Instruments  with 
high  content  validity  encompass  (measure)  a  large  portion  of  the  domain  being 
examined.  For  example,  a  test  to  examine  basic  mathematics  skills  would  have  high 
content  validity  if  it  contained  questions  concerning  addition,  subtraction, 
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multiplication,  and  division.  Such  a  test  would  have  low  content  validity,  however,  if  it 
was  to  measure  advanced  calculus  skills. 

3. 1.1. 2  CRITERION-RELATED  VALIDITY.  Criterion-related  validity  involves  an 
assessment  of  the  ability  of  an  instrument  to  predict  outcomes  in  a  specific  situation. 
Concurrent  and  predictive  validity  are  two  aspects  of  interest  of  criterion-related 
validity. 

Concurrent  validity  measures  the  agreement  between  the  instrument’s  score  or 
recommendation  and  other  established  indicators  of  the  domain  being  examined. 

Using  the  above  mathematics  example,  if  a  student  who  scored  high  on  the  test  also 
had  received  high  marks  in  mathematics  courses,  the  test  would  possess  concurrent 
validity.  Anastasi  (1982)  points  out  that  concurrent  validity  is  relevant  in  diagnosis  of 
existing  status  rather  than  prediction  of  future  ability. 

Concurrent  validity  is  often  used  as  an  efficient  alternative  of  assessing  predictive 
validity.  The  time  and  cost  required  to  properly  assess  predictive  validity  is  often  high 
(such  as  in  a  hazardous  situation)  making  the  use  of  concurrent  validity  strategies  very 
attractive  (Meister,  1985).  By  comparing  the  results  of  the  test  to  existing  indicators  of 
performance,  the  degree  of  criterion-related  validity  is  easily  measured. 

Predictive  validity  focuses  on  the  prediction  of  future  behavior  based  on  the 
performance  measured  by  the  test  instrument.  Predictive  validity  is  of  interest  in  tests 
used  for  selection  and  classification  of  personnel.  Screening  applicants  for  pilot 
training,  or  identification  of  students  likely  to  benefit  from  a  specialized  education 
program  are  two  examples  where  predictive  validity  is  relevant. 

3. 1.1.3  CONSTRUCT  VALIDITY.  Construct  validity  is  probably  the  most  difficult 
of  the  various  validity  concepts  to  adequately  understand.  Defined,  construct  validity  is 
the  extent  to  which  an  instrument  measures  a  theoretical  construct,  or  underlying  trait. 
Sanders  and  McCormick  (1987)  consider  construct  validity  to  be  assessed  through  "the 
accumulation  of  empirical  evidence  regarding  the  measurement  in  question"  (p.  63). 
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3.1. 1.4  RELEVANCE  OF  VALIDITY  MEASURES.  Content  validity  is  an 
important  measure  in  the  evaluation  of  the  alternative  intelligence  theories  presented 
in  this  paper.  The  theory  selected  must  cover  a  large  portion  of  the  intelligence  aspects 
of  the  domain  in  question  to  allow  proper  assessment  of  the  domain  with  respect  to 
expert  system  technology  capabilities. 

Predictive  validity  is  identified  as  being  useful  for  screening  future  outcomes  and 
concurrent  validity  useful  for  determining  present  status.  The  objective  of  the  DSAT  is 
to  determine  which  problem  domains  can  be  successfully  addressed  using  expert 
systems  technology,  therefore,  concurrent  validity  is  most  relevant  to  the  tool  proposed 
in  this  study.  The  theory  selected  should  possess  concurrent  validity  when  used  in  the 
proposed  application. 

Construct  validity  is  the  most  "sought  after"  validity;  a  tool  with  construct  validity 
will  encompass  the  other  aspects  of  content  and  criterion-related  validity  (Anastasi, 
1982).  The  theory  that  is  selected  must  have  evidence  of  construct  validity  to  insure 
the  foundation  of  the  DSAT  is  sound. 

3.1.2  RELIABILITY.  Reliability  is  the  degree  of  stability  or  consistency  of  a 
measurement  across  representative  samples  or  over  time  (Sanders  and  McCormick, 

1987).  This  is  a  measure  of  the  instrument’s  freedom  from  error;  the  extent  to  which 
differences  in  test  scores  are  due  to  the  "true"  differences  between  the  tested  aspects  of 
the  subjects  (Anastasi,  1982). 

Reliability  is  often  assessed  by  administering  the  test  at  one  time,  and  then  again  at 
some  future  time.  The  correlation  of  the  two  scores  is  the  reliability  coefficient  of  the 
particular  instrument.  Sanders  and  McCormick  (1987)  report  a  reliability  coefficient 
of  0.80  or  better  is  considered  satisfactory.  The  selected  theory  should  be  relatively 
free  from  random  errors  in  application  or  evaluation  to  insure  a  highly  reliable  tool. 

3.1.3  OPERATIONAL  CONSIDERATIONS.  There  are  other  aspects  of 
importance,  besides  the  issues  of  validity  and  reliability,  when  designing  a  tool  for 


28 

practical  use.  For  this  application,  unambiguous  definition  of  the  domain  attributes, 
sensitivity  to  differences  among  similar  domains,  and  ease  of  use  are  identified  as 
relevant. 

3. 1.3.1  UMAMBIGUOUS  DEFINITION  OF  DOMAIN  ATTRIBUTES. 

Unambiguous  definition  of  the  domain  attributes,  while  related  to  content  validity, 
deserves  special  attention.  As  explained  in  the  opening  paragraphs  of  this  paper,  one 
goal  of  this  research  is  to  provide  a  tool  that  provides  definitive  recommendations  to 
decision-makers  regarding  domain  suitability.  To  provide  this  type  of 
recommendation,  the  domain  and  expert  system  attributes  must  be  specified  and 
understood,  with  little  room  for  alternative  interpretations. 

3. 1.3.2  SENSITIVITY  TO  DOMAIN  DIFFERENCES.  Sensitivity  to  differences 
among  domains  also  supports  the  goal  described  above.  Since  decision-makers  will 
often  be  considering  several  (possibly  related)  domains  for  a  particular  expert  system, 
the  ability  to  differentiate  among  them  is  critical  to  providing  useful  recommendations 
of  domain  suitability.  The  selected  theory  of  intelligence  should  provide  a  means  of 
demonstrating  domain  differences. 

3. 1.3.3  EASE-OF-USE.  Ease-of-use  a  critical  aspect  to  the  ultimate  acceptance  of 
any  measurement  tool.  Practitioners  of  knowledge  engineering  often  work  under  tight 
budget  and  time  constraints.  The  tool  that  is  easily  administered,  scored,  and 
interpreted  is  the  tool  that  is  implemented.  Tools  with  high  face  validity  (the  degree  a 
tool  appears  to  measure  what  is  supposed  to  measure)  are  generally  more  accepted  by 
users  than  those  lacking  face  validity.  Also,  compact  (parsimonious)  tools  are  highly 
valued.  Therefore,  the  theory  of  human  cognitive  task  execution  that  is  selected  should 
address  these  ease-of-use  criteria. 

3.2  APPLICATION  OF  CRITERIA  TO  INTELLIGENCE  THEORIES.  Using  the 
criteria  listed  above,  the  S-R  and  Gestalt  theories  are  immediately  discarded. 

Disregarding  the  issues  of  validity  and  reliability,  their  lack  of  structure,  inability  to 
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discriminate  between  potential  domains,  and  difficulty  of  application  to  complex 
problem  domains  makes  them  inappropriate  for  the  envisioned  tool.  This  leaves  the 
psychometric  (Guilford’s  SOI  model)  theory  and  the  information-processing 
(Sternberg's  components  of  intelligence)  theory. 

It  is  critical  to  remember  the  intended  use  of  the  theory  selected.  The  goal  of  the 
present  research  is  not  understanding  human  intellectual  functioning  per  se,  but 
specifying  the  attributes  of  the  information  and  processes  of  intelligence  used  in  a 
particular  problem  domain.  This  catalog  of  attributes  can  then  be  compared  to  the 
catalog  of  attributes  for  a  particular  expert  system  (or  general  capabilities  of  expert 
systems  technology)  to  assess  the  degree  of  compatibility  between  the  two. 

While  the  information  processing  approach  is  strongly  indicated  as  relevant  to 
understanding  human  cognition  (particularly  in  terms  of  construct  validity),  it  fails  in 
several  areas  as  a  candidate  for  the  domain  analysis  tool.  Sternberg  (1977),  points  out 
the  information  processing  theories  suffer  from  lack  of  parsimony.  This  is  due  to 
lengthy  statements  of  processes,  that  interact  with  other  processes.  This  leads  to 
difficulty  in  assessing  the  contribution  individual  processes  make.  This  translates  into 
lack  of  sufficient  sensitivity  to  discriminate  between  domains. 

The  reliability  of  the  information  processing  approach  is  suspect  for  this  application. 
The  large  number  of  processes  required  in  defining  domain  intelligence  can  lead  to 
omissions  of  important  processes  (or  process  relationships)  that  may  result  in  failure  to 
adequately  capture  the  critical  aspects  of  the  domain.  The  random  error  that  could  be 
introduced  is  unacceptable  for  this  application. 

Next,  only  processes  are  accounted  for;  information  types  (the  kinds  of  information 
used  by  the  human  to  make  domain  specific  decisions  about  task  execution)  are  not 
addressed.  This  violates  the  criteria  of  sufficient  content  validity  and  sensitivity.  The 
type  of  information  is  an  important  factor  in  expert  systems  applications.  For  example, 
auditory  inputs  such  as  engine  noises  (pings,  chugs,  etc.)  are  not  directly  usable  by  a 
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computer.  An  automotive  repair  expert  system  would  fail  in  situations  where  auditory 
evaluation  is  the  primary  diagnostic  input.  A  useful  domain  analysis  tool  must  make 
distinctions  of  information  type. 

The  nature  of  the  theory  is  troublesome  for  this  application  as  well.  The  typical 
analog  of  the  information  processing  theory  is  the  digital  computer.  Since  the  goal  of 
this  research  is  the  direct  comparison  of  domain  attributes  to  expert  system  technology 
attributes,  the  existing  close  relation  of  the  information  processing  theory  to  expert 
system  functioning  may  result  in  a  "conflict  of  interest."  Information  processing  theory 
will  necessarily  cast  the  domain  of  interest  in  its  own  image;  defining  the  domain  in 
terms  that  may  not  be  appropriate  to  the  domain,  while  implying  more  similarity 
between  a  domain  and  expert  systems  than  may  exist. 

Several  researchers  believe  that  information  processing  theory  is  not  the  definitive 
theory  of  human  intelligence  and  problem  solving.  Guilford  (1985)  points  out  that 
there  is 

...  danger  of  trying  to  picture  human  information  processing  too  much  in  the 
image  of  computer  performance.  The  fact  that  the  computer  and  the  brain 
achieve  similar  results  is  not  proof  that  they  do  so  in  the  same  manner  (p.  238). 

Dreyfus  and  Dreyfus  (1986),  citing  the  inability  of  artificial  intelligence  researchers  to 

produce  humanlike  understanding  using  rules  based  on  the  mind’s  processes,  argue  "... 

the  traditional  view  (information  processing  theory)  of  mind  has  shown  itself  to  be 

inadequate."  Sternberg  (1977)  writes  that  information  processing  theory  examines  the 

components  of  intelligence  in  depth,  but  at  the  expense  of  other  possibly  useful  aspects. 

He  and  Lasaga  (1981)  build  on  that  thought,  suggesting  a  broad,  integrated  approach 

encompassing  the  most  relevant  aspects  of  both  current  and  yet-to-be  specified 

theories  as  the  best  hope  for  defining  intelligence.  Therefore,  information  processing 

theory  is  not  indicated  as  applicable  for  the  present  application. 


4.0  STRUCTURE  OF  INTELLECT  MODEL 

This  section  examines  Guilford's  Structure  of  Intellect  model  in  terms  of  the 
selection  criteria  established  in  the  previous  chapter  and  specifies  the  rationale  for 
selecting  it  over  the  competing  theories  of  human  intelligence. 

Guilford’s  SOI  model  is  the  most  logical  choice  as  a  theoretical  basis  for  examining 
domain  suitability  with  respect  to  rule- based  expert  systems  technology.  To 
summarize,  reasons  for  selecting  this  model  over  the  others  are:  sensitivity  to  many 
aspects  of  a  domain  and  unambiguous  representation  of  those  aspects  (over  150 
attributes  can  be  specified);  the  specificity  of  the  SOI  model  will  enhance  the  reliability 
of  the  tool;  the  SOI  model  is  easy  to  understand  (practitioners  can  intuitively  grasp  its 
focus);  the  SOI  model  has  high  content  validity;  and  the  SOI  model  has  been 
developed  over  the  past  40  years,  resulting  in  several  refinements  and  an 
"accumulation  of  empirical  evidence"  suggesting  its  construct  validity  for  this  approach. 

4.1  CONTENT  VALIDITY.  The  SOI  model  meets  this  criterion  since  the  model 
examines  over  150  facets  of  a  domain.  Specification  of  a  particular  operator,  content, 
and  product  specifies  the  type  of  process,  the  type  of  information  used,  and  the  type  of 
information  produced  during  the  activity.  This  model  allows  sufficient  domain 
coverage  to  establish  a  high  degree  of  satisfactory  content  validity. 

4.2  CRITERION- RELATED  VALIDITY.  Both  predictive  and  concurrent  validities 
have  been  established  for  the  SOI  model  in  many  different  intelligence-  and  cognition- 
related  applications  such  as  intelligence  testing,  creativity  measurement,  and  job 
selection  (Guilford,  1985;  Guilford  and  Hoepfner,  1971;  Meeker,  1969).  While  the 
theory  does  possess  satisfactory  levels  of  criterion-related  validity,  the  specific 
criterion-related  validity  for  this  application  must  be  established. 
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4.3  CONSTRUCT  VALIDITY.  While  the  theory  has  existed  for  over  three  decades, 
it  remains  a  viable  tool  for  understanding  the  various  aspects  of  human  intelligence. 

Kolodner  (1984)  posits  the  structure  of  the  information  used  in  human  intellectual 
problem  solving  is  different  in  experts  than  in  novices,  based  on  her  research.  The 
importance  of  information  structure  cited  by  Kolodner  in  the  mind  of  the  problem 
solver  suggests  that  a  structural  approach,  like  the  SOI  model,  is  useful  in  describing 
application  domains.  This  lends  some  support  to  the  existence  of  construct  validity  in 
this  theory. 

The  SOI  model  has  been  used  as  a  job  analysis  tool,  which  is  similar  to  the 
envisioned  DSAT.  As  cited  above,  Bringelson,  et  al.,  (1987)  suggested  using  elements 
of  job  analysis  in  the  evaluation  of  candidate  domains.  This  observed  correlation  with 
other  types  of  tests  is  also  supportive  of  construct  validity  (Anastasi,  1982). 

4.4  RELIABILITY.  The  layout  of  the  SOI  model  and  the  terms  used  lend 
themselves  to  a  reliable  method  of  describing  application  domains  and  expert  systems 
technology  attributes.  The  development  of  logical  and  specific  administration 
procedures  will  further  insure  the  reliability  of  the  instrument.  Section  7.3  examines 
the  results  of  a  study  conducted  to  assess  the  reliability  of  the  developed  tool. 

4.5  OPERATIONAL  CONSIDERATIONS.  The  nature  of  the  SOI  model  lends 
itself  to  unambiguous  definition  of  the  various  attributes  under  consideration  in  a 
domain.  The  vast  majority  of  all  attributes  composing  the  SOI  model  have  been 
operationally  defined,  significantly  reducing  the  possibility  of  mis-identifying  an 
attribute  (Guilford,  1985).  This  characteristic  is  also  useful  in  establishing  the 
sensitivity  of  the  theory.  The  significant  number  of  attributes  contributes  to  the  ability 
to  specify  differences  between  domains. 

This  theory  also  appears  to  lend  itself  towards  ease  of  use.  The  three  dimensional 
layout  of  the  model  appears  to  be  well  suited  for  the  desired  tool,  both  in  terms  of 
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administration  and  evaluation.  The  high  face  validity  of  the  SOI  model,  which  will  be 
evident  to  practitioners,  is  another  positive  aspect  that  meets  the  ease-of-use  criteria. 


5.0  STATEMENT  OF  OBJECTIVES 

Having  selected  a  suitable  theory  of  human  cognition  the  objectives  of  this  research 
can  now  be  stated:  The  primary  objective  is  to  apply  theory  to  achieve  a  tool  capable 
of  adequately  describing  the  information  requirements  of  a  domain  to  allow  the 
comparison  of  the  domain  information  requirements  with  the  capabilities  of  expert 
systems  technology.  Concurrent  with  this  objective  is  presentation  of  the  collected  data 
in  a  manner  which  effectively  conveys  the  significant  information  elements  of  the 
domain. 

A  secondary  objective  is  the  development  of  a  preliminary  means  of  combining  the 
elicited  domain  information  with  the  capabilities  of  expert  systems  technology  to 
produce  a  recommendation  of  domain  suitability.  Also,  an  assessment  of  the  reliability 
of  the  tool  for  analyzing  domains  will  be  undertaken.  Finally,  areas  of  future  work  for 
addressing  the  above  goals  will  be  discussed. 
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6.0  DEVELOPMENT  OF  THE  DOMAIN  SUITABILITY  ANALYSIS  TOOL 

This  section  presents  the  rationale  and  development  of  procedures  used  to  apply 
Guilford’s  SOI  model  to  the  task  of  defining  a  domain  with  reference  to  expert  systems 
technology  capabilities. 

The  SOI  model  offers  a  particularly  useful  means  for  quickly  and  efficiently 
examining  a  domain.  This  aspect  is  called  the  "psychoepistemology"  by  Guilford 
(1967).  Thirty  categories,  termed  content-product  features  are  identified  in  the  SOI 
model  of  fundamental  characteristics  of  information,  by  examining  a  domain  using  this 
psychoepistemology,  the  basic  information  requirements  of  the  domain  can  be 
captured. 

Once  the  basic  information  elements  of  the  domain  have  been  described,  the 
operations  performed  upon  those  elements  can  be  identified.  Guilford  has  selected 
five  mental  operations  as  relevant.  These  five  elements  compose  the  operator 
dimension  of  the  model.  Once  the  operators  relevant  to  the  domain  in  question  are 
specified,  the  domain  is  defined  in  a  manner  suitable  for  comparison  with  the 
capabilities  and  limitations  of  expert  systems  technology. 

A  similar  approach  can  be  taken  to  specify  the  capabilities  of  expert  systems 
technology.  Using  the  SOI  model,  the  range  of  information  inputs  and  outputs 
supported  by  expert  systems  technology  can  be  identified.  The  operations  performed 
by  the  expert  system  can  also  be  identified.  The  domain  and  the  current  state  of  expert 
systems  technology  can  now  be  described  in  terms  of  a  common  denominator,  the  SOI 
model,  facilitating  their  comparison. 

By  examining  the  degree  of  fit  between  critical  areas  of  the  domain  and  the  same 
areas  supported  by  the  expert  systems  technology,  it  is  possible  to  make  inferences 
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regarding  the  degree  of  domain  suitability  for  an  expert  systems  solution.  Also, 
consideration  of  the  frequency  of  occurrence  of  both  information  elements  and 
required  operators  allow  examination  of  the  impact  of  a  specific  item  on  the  overall 
suitability  of  the  domain.  For  example,  if  a  domain  occasionally  required  processing  of 
visual  information  and  expert  systems  technology  could  only  perform  visual  processing 
in  a  limited  manner,  but  otherwise  met  the  all  of  the  domain’s  requirements,  the 
decision-makers  could  then  consider  other  alternatives,  such  as  allocating  visual 
processing  to  a  human  operator. 

By  providing  decision-makers  with  relevant  information  about  the  degree  and 
nature  of  the  domain/expert  systems  compatibility,  allocation  issues  between  man  and 
machine  can  be  addressed  and  the  development  of  partial  solutions  utilizing  an  expert 
systems  approach  can  still  be  pursued.  This  information  will  prove  extremely  valuable 
since  it  will  permit  the  decision-makers  to  see  the  effect  of  implementing  alternative 
solutions  in  terms  of  the  information  requirements  of  the  domain. 

6.1  WORKING  HYPOTHESES  FOR  DEVELOPING  THE  DSAT.  In  developing 
these  procedures  for  evaluating  domain  suitability  for  expert  systems  technology,  the 
primary  assumption  is  the  information  requirements  (that  is,  the  kinds  of  information 
used  to  make  task  execution  decisions  and  the  processes  used  in  making  those 
decisions)  of  the  domain  dictate  its  suitability  for  expert  systems  solution.  Human 
beings,  who  act  in  domains,  function  successfully  in  those  domains  using  information 
organized  as  knowledge.  The  underlying  organization  (structure)  of  that  information 
can  be  discovered  by  specifying  the  relevant  information  elements  of  the  domain. 
Unfortunately,  no  universal  taxonomies  of  domain  information  requirements  exist  for 
any  problem  domain.  We  therefore  must  query  the  domain  expert  in  an  attempt  to 
understand  the  domain’s  requirements. 

The  DSAT  is  designed  to  describe  the  domain  information  elements  (based  on  the 
SOI  model)  as  revealed  by  the  domain  expert  and  then  produce  a  recommendation 


37 


based  on  the  degree  of  support  provided  by  expert  systems  technology  for  those 
information  elements.  While  some  researchers  have  attempted  (with  little  success)  to 
map  domains  in  terms  of  expert  systems  capabilities,  we  define  expc.i  systems 
suitability  in  terms  of  the  domain’s  information  requirements. 

Therefore,  Hypothesis  1  states: 

The  DSAT  will  provide  a  description  of  the  information  types  and  processes  used  for 
performance  of  domain  tasks  by  the  human  expert. 

Another  assumption  regarding  tool’s  development  (as  described  earlier  in  Section 

3.0)  is  that  for  the  DSAT  to  be  useful,  it  must  be  incorporate  both  the  functional 

aspects  of  validity  and  reliability,  and  be  easily  implemented  and  interpreted. 

Practitioners  will  not  use  a  tool  that  is  cumbersome  and  difficult  to  understand,  no 

matter  how  valid  and  reliable  its  developers  claim  it  to  be. 

Since  the  SOI  model  includes  150  different  components  (and  Guilford  hypothesizes 

even  more),  domain  evaluation  requiring  a  question  for  each  component  could  prove 

tedious  for  all  but  the  most  highly  motivated  people.  Therefore,  pursuing  a  method  of 

"pruning  the  tree  of  possibilities"  to  reduce  the  analysis  to  a  manageable  size,  but  still 

retaining  the  power  of  the  SOI  approach,  is  of  high  priority.  Hypothesis  2,  then,  states: 

The  DSAT  will  be  easily  administered,  analyzed,  and  interpreted. 

Hypothesis  3  states: 

The  DSAT  will  possess  a  high  degree  of  reliability  for  describing  domain  information 
requirements. 

Hypothesis  4  states: 

The  DSAT  will  possess  a  high  degree  of  validity  for  describing  domain  information 
requirements. 

6.2  ORGANIZATION  OF  THE  DSAT.  The  basic  procedure  involves  a  series  of 
questions  about  the  domain  of  interest  regarding  the  various  dimensions  of  the  SOI 
model.  The  DSAT  has  three  sections:  Part  I  examines  the  structural  components 
based  on  the  content-product  aspects  of  the  domain;  Part  II  exaines  the  operational 
components,  which  incorporates  the  domain  operators  with  element  frequency. 
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criticality  and  difficulty;  and  Part  III,  which  presents  the  domain  attributes  in  four 
levels  of  abstraction,  leading  to  the  domain  suitability  index  which  describes  the  overall 
domain  suitability  for  an  expert  systems  technology  solution. 

6.2.1  ASSESSING  THE  STRUCTURAL  COMPONENTS  OF  THE  DOMAIN.  This 
is  the  first  step  in  the  DSAT  process.  By  using  the  content-product  dimensions  of  the 
SOI,  a  30  element  psychoepistemology  is  described  which  provides  a  means  for 
determining  the  basic  information  elements  evident  in  the  domain.  By  having  the 
domain  expert  indicate  the  presence  or  absence  of  each  component,  the  structural 
information  requirements  for  the  domain  are  established. 

This  simple  yes/no  procedure  reduces  the  sample  space  of  possible  SOI 
components  by  five  with  each  "no"  answer.  For  example,  with  150  possible  components 
(5  contents  x  6  products  x  5  operators),  any  content-product  element  not  in  the  domain, 
eliminates  five  possible  operators  that  could  have  acted  upon  it  (see  Figure  3).  Also, 
we  defer  decisions  regarding  component  frequency,  criticality,  and  difficulty  until  the 
second  step  of  the  procedure,  reducing  the  number  of  components  (and  time  required 
for  the  interview)  to  be  considered  by  the  domain  expert. 

6.2.2  ASSESSING  THE  OPERATIONAL  COMPONENTS  OF  THE  DOMAIN. 

The  second  phase  of  the  DSAT  administration  involves  determining  the  relevant 
operators  (mental  functions)  required  to  act  on  the  information  elements  defined  in 

the  previous  step,  and  to  establish  the  criticality,  frequency  and  difficulty  of  the  various 
components  to  allow  comparison  with  expert  systems  technology  attributes.  As  an 
example,  assume  10  of  the  30  content-product  elements  are  identified  as  domain¬ 
relevant  in  Part  I.  In  Part  II,  the  domain  expert  identifies  which  operators  are  relevant 
for  each  of  the  10  elements  and  the  degree  of  frequency,  criticality,  and  difficulty 
associated  with  each. 

The  selection  of  the  operator  associated  with  the  component  is  essential,  since  the 
operators  (representative  of  mental  activities)  are  generally  the  weakest  links  in  expert 


system  implementations,  and  the  most  significant  contributions  of  the  human  to  goal 
achievement.  Their  implementation  is  "weak"  in  the  sense  that  researchers  are  not  yet 
able  to  simulate  the  robust  and  flexible  nature  of  most  mental  processes. 

Understanding  the  recurrence,  importance,  and  level  of  context  complexity  of  a 
particular  operator  within  a  domain  becomes  the  foundation  for  decisions  regarding 
supportability  of  successful  performance  using  an  expert  systems  approach. 

6.2.3  DIMENSIONS  OF  INTEREST.  There  are  three  dimensions  of  each  domain 
information  element  that  are  essential  to  understanding  the  impact  of  that  element  on 
domain  performance  and  expert  systems  supportability.  These  three  dimensions  are 
frequency,  criticality,  and  difficulty.  When  preparing  any  type  of  analysis  tool,  be  it  a 
task  analysis,  job  analysis,  or  other,  the  frequency  of  occurrence  and  criticality  of  the 
particular  item  are  generally  examined.  These  two  measures  are  reliable  indicators  of 
the  significance  of  the  attribute  to  the  overall  system  being  analyzed.  Estimates  of 
frequency  and  criticality  of  an  attribute  provide  a  basis  for  decisions  regarding  the 
impact  of  the  attribute  on  performance. 

Item  difficulty  is  an  indicator  of  the  complexity  and  ambiguity  inherent  in  the 
domain  context.  Within  DSAT,  each  domain  information  element  question  provides 
generic  examples  of  three  levels  (low,  moderate,  and  high)  of  difficulty  for  that 
element.  These  examples  serve  as  anchors  on  the  rating  scale.  The  domain  expert 
selects  the  example  that  most  closely  approximates  activities  in  his  domain,  which 
provides  an  estimate  of  the  complexity  and  ambiguity  of  the  information  context  for 
that  information  element.  Therefore,  item  difficulty  can  possess  one  of  three  values. 

In  the  DSAT,  a  five-point  Likert  rating  scale  is  used  to  evaluate  the  frequency  and 
criticality  of  separate  information  components.  Ranging  from  1  (highly  infrequent  or 
highly  noncritical)  to  5  (highly  frequent  or  highly  critical),  the  scale  allows  the  domain 
expert  to  provide  a  rating  of  the  information  component  relative  to  its  occurrence  and 
impact  on  goal  achievement.  Items  that  are  more  frequent  and  critical  will  have  higher 
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scores  than  those  items  that  are  less  frequent  and  critical.  Higher  scores  imply  higher 
significance  in  achieving  successful  outcomes  in  the  domain.  Refer  to  Appendix  A  for 
a  description  of  the  DSAT  questionnaire. 


7.0  METHODS  AND  RESULTS 

This  section  presents  the  results  of  this  effort:  the  method  of  data  presentation,  the 
domain  suitability  index  calculation  algorithm  and  scale,  and  the  analysis  of  the 
reliability  of  the  DSAT  questionnaire. 

7.1  DOMAIN  DATA  PRESENTATION.  The  DSAT  gathers  a  large  volume  of 
complex  information  that  cannot  be  easily  assimilated  in  its  raw  form  by  the  domain 
investigator.  Therefore,  a  suitable  means  of  organizing  and  displaying  the  data  to 
convey  meaningful  information  and  allow  interpretations  of  the  salient  information 
elements  of  the  domain  was  developed.  Appendices  B1  and  B2  contain  the  entire 
output  generated  for  the  domain  of  debugging  a  non-complex,  small  computer 
program. 

7.1.1  ORGANIZATION.  The  dominant  emphasis  of  the  data  reduction  strategy 
employed  centers  around  combining  related  elements  to  provide  the  domain 
investigator  with  useful  information  about  the  nature  of  the  domain.  Therefore,  the 
data  gathered  by  DSAT  is  organized  in  increasingly  higher  levels  of  abstraction  which 
provide  a  corresponding  level  of  domain  understanding. 

7. 1.1.1  BENEFITS  OF  DATA  REDUCTION.  The  benefit  of  reducing  data  is  to  a 
make  a  large  volume  of  data  more  suitable  for  interpretation.  Simply  looking  at  the 
Level  0  data  matrix  provides  little  insight  into  the  overall  focus  of  a  domain.  The  Level 
1  graphs  provide  some  indication  of  the  more  important  information  elements 
(contents)  required  by  each  of  the  domain  operators.  The  Level  2  graph  provides  a 
more  focused  view  of  all  the  domain  information  elements  grouped  by  the  relevant 
operators  involved  in  the  domain.  The  Level  3  graph  depicts  an  overall  index  for  each 
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operator  and  provides  a  single  rating  describing  the  applicability  of  that  domain  to 
techniques  of  expert  systems. 

7.1. 1 .2  COSTS  OF  DATA  REDUCTION.  The  ratings  gathered  by  the  DSAT  are 
simply  averaged  based  on  the  number  of  information  elements  selected  to  arrive  at  the 
Level  1  and  2  graphs.  Averaging,  however,  loses  some  of  the  information  inherent  in 
the  raw  data  (at  this  time,  simple  averages  of  the  specified  information  elements  have 
been  selected  pending  further  investigation  of  possible  weighting  algorithms  or  other 
strategies  of  reducing  the  data).  We  have  attempted  to  mitigate  this  loss  of 
information  by  providing  the  Level  0  data  matrix  and  by  presenting  the  number  of 
information  elements  comprising  the  ratings  depicted  in  the  Level  1  and  2  graphs.  If 
the  domain  investigator  questions  any  Level  1  or  2  presentation,  he  or  she  can  review 
the  Level  0  data  matrix  to  ascertain  the  exact  information  elements  composing  the 
graph  in  question. 

7. 1.1. 3  LEVELS  OF  ANALYSIS.  In  analyzing  the  data  gathered  by  DSAT,  four 
levels  of  analysis  are  used.  Level  0  data  is  simply  the  domain  information  element 
ratings  of  frequency,  criticality,  and  difficulty  converted  to  a  common  10-point  scale. 

Level  0  data  are  classified  by  both  operator  and  content-product  categories.  Included 
in  the  Level  0  presentation  is  th  '.  number  of  elements  specified  for  each  content 
category.  Appendix  B1  depicts  the  Level  0  data  matrix  for  the  debugging  domain 
example. 

The  Level  1  analysis  averages  the  frequency,  criticality,  and  difficulty  ratings 
individually  across  products  for  each  content  category  for  all  specified  information 
elements.  This  data  is  then  graphed  for  each  relevant  domain  operator  (see  Appendix 
B2  for  Level  1  presentations).  This  allows  the  domain  investigator  to  examine  the 
relative  importance  of  each  content  category  to  the  particular  operator  in  the  domain. 

The  Level  2  analysis  presents  the  averaged  frequency,  criticality,  and  difficulty 
ratings  across  all  content-products  for  a  specific  operator  in  the  domain  (see  Appendix 
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B2).  This  presentation  allows  the  domain  investigator  to  focus  on  the  operators  critical 
to  domain  performance  and  to  understand  the  associated  frequency  and  difficulty  of 
those  operators.  The  number  of  relevant  information  elements  for  each  operator  (with 
a  possible  total  of  30)  is  also  displayed  to  permit  inferences  regarding  the  extent  of  the 
operator’s  impact  on  the  domain. 

The  Level  3  analysis  combines  the  ratings  of  operator  frequency,  criticality  and 
difficulty  with  the  number  of  elements  indicated  to  arrive  at  an  overall  operator  rating 
for  the  domain.  These  operator  ratings  are  then  combined  to  obtain  the  domain 
suitability  index  which  describes  the  degree  of  compatibility  of  expert  systems 
technology  and  the  information  requirements  of  the  domain.  Section  7.2  describes  the 
rationale  for  each  factor  of  the  domain  suitability  index. 

7.1.2  DSAT  IMPLEMENTATION.  This  section  describes  the  procedures  and 
results  obtained  (Appendices  B1  and  B2)  for  the  domain  of  debugging  a  non-complex, 
small  computer  program.  This  domain  was  selected  because  it  is  highly  constrained 
and  the  applicability  of  expert  systems  has  already  been  established.  The  goal  was  to 
determine  if  DSAT  could  provide  a  domain  assessment  consistent  with  established 
knowledge  of  the  domain  information  elements  in  order  to  establish  some  degree  of 
concurrent  validity. 

The  DSAT  was  administered  to  a  domain  expert  (nine  years  of  software 
programming  and  debugging  experience)  following  the  directions  provided  in  the 
DSAT.  The  experimenter  played  the  role  of  knowledge  engineer  and  explained  the 
background  and  intended  use  of  the  DSAT.  After  the  domain  expert  completed  Part  I 
of  the  DSAT,  the  experimenter  deleted  all  questions  regarding  the  non-applicable 
elements  in  Part  II.  The  domain  expert  then  completed  the  ratings  required  in  Part  II. 
The  entire  interview  process  lasted  approximately  75  minutes. 

Appendix  B1  describes  the  Level  0,  2,  and  3  Data  Summaries.  The  Level  0  data 
matrix  immediately  reveals  that  auditory  and  behavioral  information  is  not  included  in 
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this  domain.  Also  of  interest  is  the  relatively  equal  inclusion  of  all  operators  in  the 
domain  in  terms  of  number  of  information  elements  (57  total  out  of  a  possible  150) 
specified. 

The  Level  1  graph  (shown  in  Appendix  B2)  for  the  cognition  operator  describes  the 
data  for  13  information  elements.  The  visual  and  symbolic  contents  appear  to  be  the 
most  critical  and  frequent  for  this  operator.  The  cognition  operator  primarily  deals 
with  discovery  and  recognition  of  various  forms  of  information  (Meeker,  1969).  The 
combined  emphasis  on  the  visual  and  symbolic  contents  are  indicative  of  how  (using 
the  eyes)  domain  experts  gather  (primarily  symbolic)  domain  information.  The  high 
criticality  of  the  visual  content  supports  this  interpretation,  since  this  is  the  primary 
means  of  information  input  for  humans.  An  exit  interview  with  the  domain  expert 
confirmed  this  assumption. 

The  Level  1  graph  for  the  memory  operator  describes  a  fairly  even,  fairly  high 
(relative  to  other  operators)  emphasis  on  both  the  symbolic  and  semantic  information 
contents.  Memory,  which  is  the  storage  or  retention  of  information  for  later  use 
(Meeker,  1969),  in  this  domain  is  only  concerned  with  the  impact  of  particular  symbols 
used  and  the  meaning  implied  by  those  symbols.  The  elements,  while  fairly  critical  in 
nature  are  of  a  low  difficulty  level  (which  is  consistent  for  all  operators). 

The  Level  1  graph  for  the  divergent  production  operator  describes  again,  a  fairly 
equal,  but  low  emphasis  on  the  symbolic  and  semantic  contents  of  the  domain.  The 
lower  criticality  and  frequency  ratings  indicate  that  the  divergent  production  operator, 
which  is  generally  associated  with  creative  thought  (Guilford,  1967),  is  not  of  high 
importance  to  this  domain. 

The  Level  1  graph  for  the  convergent  production  operator  describes  a  similarly  low, 
but  slightly  greater  overall  emphasis  on  the  symbolic  and  semantic  contents  than  the 
divergent  production  operator.  This  suggests  a  slightly  greater  reliance  on  rules  and 
proceduralized  or  systematic  solution  of  problems.  This  is  expected  in  a  domain  where 


specific  faults  (bugs)  often  have  specific  remedies.  The  criticality  and  frequency  of 
occurrence  for  this  operation  are  moderately  high  (relative  to  other  operators) 
suggesting  some  reliance  on  this  operation.  Based  on  the  graphic  representation  of  her 
responses,  the  expert  has  perceived  rule- based  solution  strategies  to  be  of  more  value 
than  creativity  for  this  domain. 

The  Level  1  graph  for  the  evaluation  operator  describes  a  fairly  high  emphasis  on 
symbolic  contents  with  less  on  semantic  contents.  Again,  this  is  expected  since  non¬ 
complex,  well-defined  computer  software  faults  would  tend  toward  errors  of  form 
(spelling,  syntax,  etc.)  than  errors  of  content  (use  of  procedures,  improper  design  of  the 
software  relative  to  the  program  goal,  etc.).  The  evaluation  operator  appears  to  be  of 
moderate  significance  in  this  domain. 

The  Level  2  graph  for  the  57  domain  elements  indicates  a  relatively  higher  domain 
criticality  for  cognition  (the  discovery  or  recognition  of  factors)  which  appears 
intuitively  correct  in  a  domain  where  discovering  errors  is  of  high  priority.  The 
memory  operation  is  secondary  in  importance  to  cognition  based  on  overall  criticality 
and  appears  less  frequently.  Evaluation  and  convergent  production  are  very  similar, 
and  while  their  frequencies  and  levels  of  difficulty  do  differ,  there  is  little  overall 
difference.  Divergent  production  appears  to  be  the  least  important  to  successful 
performance  relative  to  the  other  operators  in  this  domain. 

Based  on  the  Level  2  data,  the  expert’s  decreasing  focus  on  the  mental  operators  is 
in  the  following  order:  recognition  (finding  a  possible  bug),  recall  (storing  and/or 
recalling  the  possible  bug  at  the  right  time),  and  evaluation  (is  this  really  a  bug?)  and 
rule  selection  (now  how  do  I  fix  it?)  all  at  fairly  low  levels  of  difficulty  appear  to  be 
intuitively  correct  for  this  domain.  Overall,  the  information  presented  by  DSAT 
appears  to  be  highly  correlated  with  the  established  understanding  of  the  activities  and 
information  requirements  of  the  debugging  domain.  Therefore,  for  this  domain  and 
level  of  DSAT  refinement,  these  results  indicate  a  fair  degree  of  concurrent  validity. 
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7.2  DOMAIN  SUITABILITY  INDEX.  The  Domain  Suitability  Index  (DSI) 
correlates  the  domain  information  requirements  with  the  capabilities  of  expert  systems 
technology.  By  weighting  the  various  dimensions  and  domain  operators  based  on  the 
relative  supportability  offered  by  expert  systems  technology,  it  may  be  possible  to 
describe  the  domain  with  a  single  rating  that  can  be  compared  to  other  domains,  or 
used  to  estimate  the  success  of  adopting  an  expert  system  solution  for  the  domain. 

7.2.1  RATIONALE  FOR  DIMENSION  WEIGHTING  FACTORS.  The  weighting 
factors  used  for  this  analysis  are  only  gross  estimates  of  the  possible  "true"  weights 
needed  to  accurately  describe  the  supportability  of  expert  systems  and  are 
implemented  strictly  as  an  exploratory  approach.  Future  research  must  experimentally 
determine  the  true  weighting  factors. 

There  are  two  sets  of  weighting  factors  used:  the  dimensions  (frequency,  criticality, 
difficulty,  and  number  of  elements)  and  the  operators  (cognition,  memory,  divergent 
production,  convergent  production,  and  evaluation).  The  dimension  factors  will 
examined  first. 

7.2.1. 1  DIMENSION  FACTOR  WEIGHTS  .  These  factor  weights  (which  sum  to 
1 .0)  translate  the  impact  of  information  elements  based  on  the  dimension  ratings 
collected  by  the  DSAT.  These  weights  are  established  by  the  degree  of  influence 
imparted  by  the  dimension  on  the  development  of  an  expert  system  and  the  impact  of 
the  dimension  in  successful  domain  performance. 

The  dimension  of  difficulty  defines  the  degree  of  ambiguity  and  complexity  of  an 
information  element  context  and  carries  a  weighting  factor  of  0.50.  This  is  the  highest 
weighted  factor  since  this  dimension  is  appears  strongly  correlated  with  high-risk 
expert  systems  implementations.  That  is,  elements  with  high  difficulty  ratings  are  often 
not  supported  by  expert  systems  techniques. 

The  next  highest  rated  factor  is  criticality.  Since  this  dimension  indicates  the 
relative  importance  of  the  information  element  to  the  domain,  elements  with  high 


ratings  should  be  included  to  insure  successful  domain  performance.  A  factor  of  0.35  is 
assigned  to  this  dimension.  The  last  two  dimensions,  frequency  and  number  of 
information  elements  specified  for  a  given  operator,  appear  to  be  of  little  importance 
in  terms  of  impact  upon  expert  systems  supportability.  Therefore,  frequency  is 
assigned  a  factor  of  0.10  and  number  of  elements  0.05. 

For  example,  a  highly  complex  and  critical,  but  infrequent,  operator  will  have  a 
greater  impact  on  domain  performance  (and  a  correspondingly  high  demand  for  expert 
systems  technology  support)  than  a  highly  frequent,  but  less  critical  and  complex 
operator.  In  summary,  Equation  1  describes  the  model  for  the  operator  combined 
rating  (OCR)  score  (which  ranges  from  0.0  to  10.0)  of  each  operator  based  on  the  four 
dimensions  weights: 

OCR  =  0.10*(F)  +  0.35*(C)  +  0.50*(D)  +  0.05*(N).  (1) 

Where  F  is  the  Level  2  Frequency  rating  for  the  domain  operator,  C  is  the  Level  2 
Criticality  rating  for  the  domain  operator,  D  is  the  level  2  Difficulty  rating  for  the 
domain  operator,  and  N  is  the  number  of  information  elements  specified  for  the 
domain  operator.  Each  of  the  frequency,  criticality,  and  difficulty  ratings  and  the 
number  of  elements  are  scaled  to  range  from  0.0  to  10.0. 

7.2. 1.2  OPERATOR  FACTOR  WEIGHTS.  The  operator  factor  weights  (which  sum 
to  1.0)  translate  the  support  provided  by  expert  systems  technology  for  each  operator  in 
the  domain.  The  sum  of  the  weighted  factors  of  the  dimensions  (each  of  the  Operator 
Combined  Rating  scores)  and  the  weighted-sum  of  those  OCR  scores  is  the  domain 
suitability  index  (DSI)  score  for  the  domain.  This  score  provides  a  measure  of  the 
suitability  of  the  domain  for  expert  systems  technology.  The  following  paragraphs 
explore  the  rationale  for  the  weights  of  each  OCR  score. 

The  operator  of  divergent  production  is  highly  correlated  with  creativity  and 
original  thought,  areas  which  are  largely  unsupported  jy  expert  systems  technology. 
Therefore,  this  domain  operator  will  have  the  greatest  impact  of  all  domain  operators 
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relevant  to  suitability  for  expert  systems  technology.  To  reflect  this  in  the  DSI  score, 
the  divergent  production  OCR  score  has  a  weight  of  0.50. 

The  cognition  operator  deals  with  discovery  and  recognition  of  information.  While 
some  progress  in  pattern  recognition  and  other  machine  "discovery"  like  functions  have 
been  developed,  they  are,  at  present,  task  specific  and  suggest  a  low-level  of  expert 
systems  technology  support.  This  OCR  score  carries  a  corresponding  weight  of  0.25  to 
indicate  this  questionable  degree  of  support. 

The  evaluation  operator  is  concerned  with  comparing  and  judging  information 
against  a  standard  or  each  other.  To  the  extent  that  the  rules  for  the  comparison  can 
be  specified  and  the  type  of  information  being  evaluated  is  suitable  for  machine 
analysis,  this  operator  can  be  supported  using  an  expert  systems  approach.  Therefore, 
the  OCR  score  for  evaluation  carries  a  weight  of  0.15. 

Memory,  which  involves  storage  and  retrieval  of  information  appears  to  be  well 
supported  by  expert  systems  technology.  While  an  important  aspect  of  the  memory 
operator  is  learning,  or  acquiring  new  knowledge  (which  is  weakly  supported  by  expert 
systems  technology)  it  could  argued  this  operator  should  carry  a  fairly  high  weighting 
factor.  However,  for  this  preliminary  analysis,  the  memory  OCR  score  will  be  carry  a 
weighting  factor  of  0.05. 

Convergent  production,  the  operator  that  deals  with  following  a  set  of  rules  to 
arrive  at  an  answer,  is  viewed  as  being  highly  similar  to  the  functionality  of  a  rule-based 
expert  system.  Because  of  this  functional  similarity,  the  OCR  score  for  this  operator  is 
also  factored  at  0.05. 

Equation  2  summarizes  the  model  of  the  Domain  Suitability  Index  (DSI)  score: 

DSI  =  0.25*(C  OCR)  +  0.05*(M  OCR)  +  0.50*(DP  OCR)  +  (2) 

0.05*(CP  OCR)  +  0. 15*(E  OCR) 

Where  the  value  in  parentheses  is  the  relevant  operator  combined  rating  (OCR)  score 
described  in  the  preceding  section. 
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7.2.2  ESTABLISHMENT  OF  DOMAIN  SUITABILITY  INDEX  ANCHORS.  To 
determine  whether  the  DSAT  questionnaire  can  provide  an  accurate  estimate  of 
suitability  for  specific  domains,  it  is  necessary  to  select  at  least  two  domains  to  serve  as 
"anchors"  to  establish  endpoints  for  the  domain  suitability  index  score  (DSI).  The 
anchor  domains  are  selected  based  on  existing  knowledge  of  the  domain  suitability  to 
expert  systems  technology.  The  domain  selected  for  the  lower  anchor  should  be  well- 
within  the  capabilities  of  expert  systems  technology  and  have  several  successful 
examples  of  expert  systems  in  existence.  The  upper  anchor  domain  should  obviously 
be  unsuitable  for  an  expert  system,  perhaps  requiring  extensive  understanding  and 
creativity  components  for  success. 

Once  the  anchor  domains  are  selected,  the  domain  profile  and  accompanying  DSI 
score  can  then  be  used  to  establish  limits  for  the  expected  range  of  DSI  scores  for  other 
domains.  Finally,  a  domain  is  tested  which  should  fall  within  the  range  provided  by 
these  two  anchors.  If  the  DSAT  questionnaire  is  performing  as  desired,  it  should 
produce  a  domain  profile  and  DSI  score  somewhere  in  between  the  domain  anchors. 

7.2.2. 1  LOWER-BOUND  DOMAIN  ANCHOR.  The  domain  chosen  for  the  lower- 
bound  anchor  was  debugging  a  non-complex,  small  (less  than  200  lines  of  code) 
computer  program  with  a  specified  function  and  inputs.  Section  7.1.2  describes  the 
rationale  for  choosing  this  domain  and  one  domain  expert’s  DSAT  results.  To 
summarize,  the  information  requirements  of  this  domain  are  fairly  well  known  and 
several  expert  systems  have  been  designed  to  perform  the  task  (Frenkel,  1985)  making 
this  domain  suitable  as  a  lower  bound  anchor. 

7. 2. 2. 2  UPPER-BOUND  DOMAIN  ANCHOR.  To  be  suitable  as  an  upper-bound 
anchor  for  the  DSI  scale,  a  candidate  domain  requires  extensive,  possibly 
inarticulatable  (deep  as  opposed  to  shallow)  domain  knowledge  for  achieving  adequate 
performance.  Other  desirable  features  include  a  high  degree  of  human  interaction 
involved  in  data  gathering,  unambiguous  and  indeterminate  levels  of  information  used 
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to  make  decisions,  and  a  high  degree  of  creativity  or  diverse  alternatives  required  for 
high  domain  performance. 

This  short  list  should  by  no  means  be  considered  exhaustive,  simply  features  of  the 
nature  of  a  domain  that  would  be  an  extremely  high  risk  venture  from  an  expert 
systems  perspective.  For  example,  managing  a  large,  multinational  corporation, 
symphonic  quality  music  composition,  writing  a  novel,  or  coaching  a  football  team  are 
all  examples  where  an  expert  system  development  program  would  probably  have  a  high 
risk  of  failure. 

With  these  examples  in  mind,  the  domain  of  developing  a  research  proposal  is 
selected.  This  domain  requires  a  large  degree  of  interpersonal  data  gathering,  which 
includes  a  significant  degree  of  interpreting  behavioral  actions  to  assess  proper 
responses  on  the  part  of  the  proposal  author.  Also,  the  degree  of  creativity  required  in 
producing  a  proposal  that  is  concise,  yet  "sells"  the  proposed  research  is  often  high  for 
successful  proposals.  Successful  proposal  authors  often  demonstrate  a  deep 
understanding  not  only  of  their  particular  field,  but  of  the  proposal  process  and  the 
politics  of  "shepherding"  a  proposal  through  the  "gates"  required  for  approval. 

Appendix  Cl  presents  the  Level  0-3  Data  Summaries  Appendix  C2  presents  the  same 
data  graphically. 

It  is  obvious  from  the  Level  2  Data  Summary  graph  that  a  high  level  of  emphasis  is 
placed  on  the  divergent  production  operator  in  this  domain,  with  the  operator  level  of 
difficulty  at  the  maximum  and  criticality  near  maximum.  The  other  operators  also 
show  significantly  higher  difficulty  ratings  than  does  the  debugging  domain.  The  DSI 
score  for  this  domain  was  also  fairly  high  at  7.2.  While  a  higher  DSI  seems  justified  by 
the  data  display,  it  must  be  remembered  that  the  DSI  algorithm  is  only  a  gross  estimate 
of  what  the  true  factors  should  be.  Further  research  of  how  expert  systems  technology 
supports  each  of  this  domain  operators  is  necessary  to  develop  a  truly  accurate  DSI 


scale. 
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The  DSI  scores  for  the  two  domains  do,  however,  indicate  the  ability  of  the  DSAT 
questionnaire  to  discriminate  between  domains  of  varying  expert  systems  technology 
suitability.  Based  on  the  preliminary  nature  of  the  DSI  algorithm  and  individual 
domain  problem  solving  style  of  the  domain  experts,  the  scores  of  the  anchor  domains 
were  not  as  diverse  (ideally  the  debugging  domain  would  be  near  0.0  and  the  grant 
proposal  domain  would  be  near  10.0)  as  desired,  but  the  scores  do  provide  a  useful 
indication  that  the  DSAT  methodology  is  fundamentally  capable  of  discriminating 
between  different  domains. 

7.2.3  TESTING  AN  UNKNOWN  DOMAIN.  Having  established  the  lower-  and 
upper-bounds  of  the  DSI  scale,  a  domain  of  possible,  but  unknown  suitability  for  expert 
systems  technology  is  tested.  This  domain  should  have  elements  of  both  anchor 
domains,  but  in  an  unknown  combination.  For  example,  the  domain  should  have  some 
degree  of  creativity  associated  with  it,  but  also  rely  on  the  use  of  rules  and  have  some 
information  inputs  be  unambiguous  and  noncomplex.  Other  desirable  domain  features 
might  include  a  requirement  for  behavioral  information  or  significant  use  of  evaluation 
or  cognition  operators  using  well-defined  information. 

Using  these  selection  criteria,  the  domain  of  evaluating  the  mental  capacities  of 
children  in  grades  K,  1,  and  2  for  entry  into  a  gifted  education  curriculum  was  selected. 

The  domain  expert  has  29  years  in  primary  grade  education  and  has  been  performing 
evaluations  of  this  nature  for  5  years.  Appendix  D1  contains  the  Level  0-3  Data 
Summaries  and  Appendix  D2  has  the  graphic  display  of  the  data.  As  predicted,  the 
DSI  score  for  the  domain  was  5.2  which  falls  between  the  two  scale  anchor  domains. 

The  operators  of  cognition  and  evaluation  appear  to  most  important  in  this  domain. 

From  the  Level  2  graph  (Appendix  D2),  the  criticality  of  both  cognition  and  evaluation 
are  high  relative  to  the  other  operators.  This  appears  intuitive  since  the  primary  goals 
of  the  gifted  education  specialist  in  a  session  are  to  discover  and  recognize  traits  that 
characterize  the  student  and  then  evaluate  those  traits  against  specified  standards. 
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White  the  overall  Level  2  difficulty  is  not  extremely  high,  the  Level  1  graph  for  the 
cognition  operator  shows  a  heavy  reliance  on  auditory  and  behavioral,  as  well  as 
symbolic  information  contents.  While  symbolic  information  contents  are  supported  by 
expert  systems  technology,  interpretation  of  auditory  and  behavioral  information  is 
only  weakly  supported.  The  Level  1  display  therefore  suggests  that  perhaps  parts  of 
this  task  may  not  be  supported  with  an  expert  system. 

The  operator  of  evaluation  also  requires  a  wide  range  of  content  support.  However, 
the  lower  levels  of  difficulty  indicate  that  perhaps  a  large  part  of  those  evaluation  tasks 
could  be  supported  using  an  expert  system.  To  the  extent  the  evaluation  rules  could  be 
represented,  the  majority  of  the  evaluation  function  would  be  supported,  relieving  the 
domain  expert  to  pursue  other  functions  (such  as  cognition)  that  are  more  suited  to  the 
human’s  capabilities. 

The  preceding  analysis  shows  the  power  of  the  DSAT  approach.  By  determining 
the  relevant  domain  information  attributes,  it  is  possible  to  make  defensible  decisions 
(by  using  the  various  Levels  of  data  summary)  regarding  the  allocation  of  those 
functions  between  the  human  and  the  system,  based  on  the  strengths  and  weaknesses 
of  each.  By  allocating  these  necessary  functions  to  the  entity  most  capable  to  perform 
the  task,  the  ultimate  performance  of  the  design  is  improved. 

7.2.4  SUMMARY  OF  THE  DSI  SCALE.  Figure  4  describes  the  anchor  scores  and 
the  unknown  domain  score  for  the  DSI  scale.  As  the  DSAT  is  used  for  diverse 
domains,  the  points  along  this  scale  will  become  better  defined.  Determining  the  cut¬ 
off  points  for  varying  levels  of  expert  systems  development  risks  will  have  to  be 
established  in  future  research.  However,  the  basic  value  of  the  DSAT  concept  is 
confirmed,  with  the  appropriate  ranking  of  domains  based  on  their  established 
applicability  for  expert  systems  technology  solutions. 
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Debugging  Domain  (D) 
DSI  •  4.4 


Research  Proposal  Domain  (R) 
DSI  -  7.2 


Gifted  Child  Assessment  Domain  (G) 
DSI  -  5.2 


Figure  4.  Domain  Suitability  Index  Scale  showing  relative  positions  of  anchor  and 

unknown  domains. 
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7.3  RELIABILITY  OF  DSAT  QUESTIONS.  This  section  describes  the  procedures 
and  results  of  investigating  the  reliability  of  the  DSAT  questionnaire.  The  rationale 
for  each  aspect  of  the  investigation  is  also  described. 

Reliability  is  examined  from  the  standpoint  of  consistency  of  results  among  subjects 
of  (assumed)  similar  experience  levels  rating  the  same  task.  Based  on  this  basic 
experimental  goal,  the  following  sections  will  describe  the  measures  of  interest  for  this 
study,  the  experimental  design,  the  method  of  data  collection,  the  method  of  data 
evaluation,  and  interpretation  of  the  results  of  the  study. 

7.3.1  MEASUREMENTS  OF  INTEREST.  The  key  measurement  of  interest  in  this 
investigation  is  he  variability  among  the  ratings  made  by  subjects.  Due  to  the  ordinal 
nature  of  the  collected  data,  statistical  analyses  which  require  interval  scales  cannot  be 
performed.  Therefore,  the  standard  deviation  of  each  operator  included  in  the  domain 
analysis  is  selected  as  the  estimator  of  variability  among  ratings  from  the  domain 
experts.  Smaller  values  of  the  standard  deviation  indicate  a  higher  degree  of 
agreement  among  domain  experts  for  that  operator  with  respect  to  the  domain. 

The  DSAT  uses  a  10-point  scale  for  comparing  ratings  across  dimensions. 

Therefore,  a  standard  deviation  of  +1.0  or  less  would  be  highly  desirable,  since  68%  of 
the  ratings  for  that  element  would  fall  within  that  range.  However,  due  to  the 
preliminary  nature  of  this  study,  standard  deviations  of  ±2.0  or  less  indicate  the  DSAT 
is  making  useful  discriminations  among  the  various  domain  elements. 

A  secondary  measure  of  interest  is  the  range  of  responses  for  a  given  domain 
element.  The  interquartile  range  (the  middle  50%  of  reported  ratings  for  a  specific 
element)  is  another  indicator  of  the  degree  of  agreement  between  domain  experts 
regarding  a  particular  domain  element.  For  this  study,  an  interquartile  range  of  3.0  or 
less  has  been  selected  as  indicative  a  suitable  ’’clustering”  of  ratings. 

While  the  specific  means  of  responses  (i.e.,  the  mean  for  the  difficulty  dimension  for 
the  cognition  operator  or  the  domain  suitability  index  score)  are  of  interest  for 


measuring  the  DSATs  degree  of  content  validity  (does  the  tool  measure  what  it  is 
supposed  to  measure),  they  are  not  the  focus  of  this  evaluation.  The  preliminary 
nature  of  the  DSI  algorithm  and  the  DSAT  questionnaire  both  make  evaluations  of 
this  particular  measure  premature.  Later  research  devoted  to  establishing  the  validity 
of  the  tool  will  definitely  focus  on  aspects  of  this  nature. 

7.3.2  EXPERIMENTAL  DESIGN.  The  basic  design  of  this  analysis  is  the 
administration  of  the  DSAT  questionnaire  to  a  ten  experts  in  a  domain,  calculation  of 
the  standard  deviations  and  ranges  of  the  responses,  and  interpretation  of  those  results 
with  respect  to  the  reliability  of  the  tool  based  on  the  measures  of  interest  described  in 
the  preceding  section.  Hypothesis  3  (described  in  Section  6.1)  is  tested  in  this  analysis. 

In  this  design,  several  assumptions  are  made  regarding  the  data  (the  ratings 
collected)  and  subjects.  The  data  is  assumed  to  be  ordinal  in  nature,  limiting  the  types 
of  statistical  analyses  that  can  be  used  to  simple  univariate  descriptors  such  as  mean, 
standard  deviation,  and  skewness.  The  data  is  also  assumed  to  be  normally  distributed 
and  independent  of  the  level  of  experience  of  the  subject.  Therefore,  subjects  should 
report  the  same  basic  scores  regardless  of  their  level  experience  for  the  given  domain 
task. 

Subjects  are  assumed  to  be  homogeneous,  in  that  their  level  of  experience  with  the 
task  and  the  underlying  foundations  of  the  DSAT  tool  are  similar.  The  order  of  the 
questions  presented  in  the  DSAT  (such  as  the  contents  within  Part  I  or  the  operators  in 
Part  II)  is  also  assumed  to  have  no  effect  on  subjects’  ability  to  rate  dimensions  of 
frequency,  criticality,  difficulty,  and  number  of  elements  in  the  domain.  It  is  assumed 
that  the  wording  of  the  domain  examples  will  impact  the  subjects’  ability  to  determine 
the  difficulty  rating  and  is  a  major  focus  of  this  study. 

7.3.3  DATA  COLLECTION.  The  basic  premise  of  the  study  was  to  administer  the 
DSAT  as  it  would  be  used  in  the  field;  administered  by  a  knowledge  engineer  who 
sought  information  about  a  specific  domain  from  a  domain  expert.  Debugging  a  non- 
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complex,  small  computer  program  with  a  specific  function  was  selected  as  the  domain 
of  interest  since  this  domain  has  been  implemented  to  some  extent  (Frenkel,  1985) 
using  expert  systems  technology  (therefore  some  knowledge  of  the  domain's  suitability 
for  expert  systems  has  been  established)  and  the  availability  of  subjects  with  domain 
expertise  was  high. 

Subjects  were  selected  based  on  prior  experience  in  performing  software  debugging 
tasks.  Subjects'  experience  ranged  from  5  to  19  years,  with  a  mean  of  9.8  years.  After 
the  experimenter  explained  the  purpose  of  the  study,  the  domain  of  interest,  the 
subject’s  role,  and  answered  any  questions,  the  subject  completed  the  informed  consent 
form  (shown  in  Appendix  E). 

Part  I  of  the  DSAT  questionnaire  (see  Appendix  A)  was  first  presented  to  the 
subject.  As  described  in  Section  6.0,  Part  I  of  the  DSAT  is  provided  to  allow  the 
domain  expert  to  define  the  content-products  relevant  to  the  domain  to  reduce  the 
number  of  Part  II  questions  to  a  manageable  size  and  to  become  acquainted  with  the 
terminology  and  format  of  the  DSAT  before  actual  ratings  are  made.  As  the  subject 
completed  each  section  of  Part  I,  the  experimenter  tailored  the  Part  II  questions  by 
crossing-out  content-products  not  specified  by  the  domain  expert. 

The  domain  expert  completed  Part  II  by  evaluating  the  frequency,  criticality,  and 
level  of  difficulty  of  each  domain  relevant  operator  with  the  content-product  elements 
specified  in  Part  I.  If  a  particular  content- product  specified  in  Part  I  was  not  relevant 
for  a  specific  operator  in  Part  II,  the  domain  expert  left  the  question  blank.  All  domain 
experts  were  able  to  complete  both  Parts  I  and  II  between  75  and  110  minutes.  Upon 
completion,  subjects  were  asked  their  number  of  years  of  experience  of  computer 
programming. 

At  the  conclusion  of  each  session,  the  experimenter  entered  the  subject's  level  of 
experience  and  domain  responses  into  a  computerized  spreadsheet  program  to 
produce  the  DSAT  Level  0  Data  Summary  (see  Appendix  F).  Each  rating  of  frequency 


and  criticality  was  multiplied  by  2.0  and  difficulty  ratings  were  multiplied  by  3.33  prior 
to  data  entry.  By  using  these  factors  (2.0  for  the  5-point  scale  and  3.33  for  the  3-point 
scale)  it  was  possible  to  put  all  three  scores  on  a  comparable  (10-point)  scale  for  later 
data  analysis. 

Since  the  Level  0  represents  the  data  in  its  most  basic  (and  difficult  to  interpret) 
form,  the  DSAT  Level  2  Data  Summary  (also  shown  in  Appendix  F)  was  used  as  the 
basis  for  the  data  analysis.  While  Level  0  data  holds  the  most  information,  the  sheer 
volume  of  data  points  makes  the  data  analysis  and  interpretation  of  the  results  more 
difficult.  However,  Level  3  Data,  which  significantly  reduces  the  number  of  data  points 
to  be  analyzed,  is  based  solely  on  the  algorithm  factors  which  have  not  been  rigorously 
established.  Any  inferences  made  from  Level  3  Data  would  have  to  be  treated  with 
caution.  The  Level  2  Data,  which  averages  the  dimensions  of  frequency,  criticality,  and 
difficulty  over  the  content-products  for  each  operator,  provides  the  best  compromise 
between  parsimony  and  information  content. 

7.3.4  METHOD  OF  DATA  EVALUATION.  As  described  in  Section  7.3.1,  the  basic 
objective  of  the  study  was  to  determine  the  degree  of  variability  among  domain  experts 
for  rating  a  specific  task.  Using  the  Statistical  Analysis  Software  (SAS),  the  Level  2 
Data  for  all  10  subjects  was  analyzed  (see  Appendix  G  for  SAS  program). 

Twenty-three  data  points  for  each  subject  were  used  from  the  Level  2  Data 
Summary,  and  are  described  in  Table  6.  The  raw  data  of  Appendix  F  was  entered  into 
the  data  analysis  program.  Table  7  lists  a  summary  of  the  univariate  outputs  for  each 
of  the  variables  listed  based  on  all  10  subjects. 

7.3.5  DATA  INTERPRETATION.  The  variability  for  frequency,  criticality,  and 
difficulty  measured  by  the  standard  deviation  was  less  than  2.0  for  11  of  these  15 
variables  as  shown  in  Table  7.  The  interquartile  ranges  were  similarly  within  the 
desired  range.  This  indicates  a  fairly  high  degree  of  agreement  among  the  domain 
experts  for  these  variables.  However,  the  standard  deviations  reported  for  the 
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TABLE  6 

Summary  of  Level  2  Data  Points  Used  (*)  in  Analysis 
DIMENSIONS 


OPERATORS 

FREQ 

CRIT 

DIFF 

NUMR 

OCR 

COGNITION 

* 

* 

* 

* 

- 

MEMORY 

* 

* 

* 

* 

- 

DIVERGENT 

PRODUCTION 

* 

* 

* 

* 

- 

CONVERGENT 

PRODUCTION 

* 

* 

' 

* 

* 

- 

EVALUATION 

* 

* 

* 

* 

- 

- 

- 

- 

* 

*  s 

LEGEND:  Abbreviation  =  Dimension  (Range  of  Values) 

FPEQ  =  Frequency  Rating  (0-10) 

DIFF  =  Difficulty  Rating  (0-10) 

CL IT  =  Criticality  Rating  (0-10) 

NLMR  =  Number  of  Elements  Rated  (0-30/3) 

OCR=Operator  Combined  Rating  Score  (0-10) ,  weighted  average  of 
dimension  ratings  (See  Equation  1) . 

DSI=Domain  Suitability  Index  Score  (0-10) ,  weighted  average  of 
operator  combined  rating  scores  (See  Equation  2) . 


TABLE  7 


Summary  of  Means,  Standard  Deviations,  and  Interquartile  Ranges  for  DSAT 

Reliability  Study  (N  =  10) 


DIMENSIONS 


OPERATORS 

FREQ 

CRIT 

DIFF 

NUM 

Mn 

6.390 

7.390 

6.080 

14.000 

COGNITION  SD 

0.950 

0.823 

1.834 

1.885 

IR 

1.250 

1.575 

2.275 

2.500 

Mn 

5.550 

6.370 

5.610 

13.000 

MEMORY  SD 

1.396 

1.133 

1.978 

3.623 

IR 

1.400 

2.050 

2.225 

3.250 

Mn 

4 . 990 

6.140 

4.870 

8.500 

DIV.  PRO.  SD 

2 .489 

2.663 

2.301 

5.949 

IR 

2.125 

2.525 

3.175 

10.500 

Mn 

5.020 

5.970 

5.200 

11.700 

CON.  PRO.  SD 

1.149 

1.180 

1.938 

3.401 

IR 

1.600 

2.225 

2.975 

4.000 

Mn 

5.440 

5.860 

5.220  | 

13.200 

EVALUATION  SD 

0.530 

0.802 

2.164 

3.225 

ID 

0.500 

0.925 

2.525 

4.500 

Mn 

5.980 

1 

Mn 

60.400 

DSI  SD 

1.360 

|  NUMR  SD 

13.049 

IR 

2.175 

1 

IR 

11.000 

LEGEND:  Abbreviation  =  Term 

DIV.  PRO  =  Divergent  Production 
CON.  PRO . =  Convergent  Production 

DSI  =  Domain  Suitability  Index  Score  (See  Equations  1  and  2.) 
NUMR  =  Number  of  Elements  Selected 
Mn  =  Mean 

SD  =  Standard  Deviation 
IR  =  Interquartile-Range 
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frequency,  criticality,  and  difficulty  ratings  for  the  divergent  production  operator  were 
over  2.0.  This  indicates  some  degree  of  disagreement  between  domain  experts 
regarding  this  operator. 

Also,  the  dimension  with  the  greatest  variability  (highest  standard  deviations)  was 
the  difficulty  dimension.  The  difficulty  dimension  was  originally  predicted  to  be  the 
largest  contributor  of  variation  among  domain  experts  (see  Section  7.3.2).  It  is  possible 
that  these  examples  may  need  revision  to  reduce  the  variability  among  respondents. 

There  is  another  factor  that  may  be  responsible  for  the  large  variation  among 
domain  experts  in  the  difficulty  dimension.  During  the  course  of  the  study,  several  of 
the  overall  ratings  given  by  some  subjects  appeared  to  be  significantly  different  from 
the  rest  of  the  group.  Looking  at  Table  8  (Subject  Selected  Raw  Data  Summary),  it 
appears  that  subjects  who  reported  more  years  of  experience  tended  to  have  higher 
difficulty  ratings  and  higher  domain  suitability  index  (DSI)  scores. 

This  raw  data  suggests  that  experience  influence  how  subjects  perceive  their 
domain.  Further  correlational  and  ANOVA  analyses  of  the  data  indicates  there  is  a 
significant  effect  for  experience.  This  effect  of  experience  on  DSAT-derived  ratings  of 
difficulty  in  general  and  the  divergent  production  operator  in  particular  are  the  most 
important  (and  unexpected)  findings  of  this  research.  While  it  was  suggested  that  some 
effect  due  to  individual  problem  solving  strategies  would  be  evident  in  the  DSAT 
results,  the  strength  of  the  differences  due  to  experience  level  was  unanticipated. 

Table  9  presents  a  summary  of  the  correlation  data  used  to  assess  the  effect  of 
experience  based  on  data  taken  from  all  10  subjects.  Experience  was  found  to 
correlate  (some  at  highly  significant  levels)  with  reported  ratings  of  difficulty  in  general 
and  divergent  production  in  particular.  This  high  correlation  of  level  of  expertise  with 
the  mental  operation  of  producing  many  alternatives  for  a  given  condition  supports 
findings  by  Chiesi,  Spilich,  and  Voss  (1979).  According  to  their  research,  experts 
consistently  generated  more  and  better 
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TABLE  8 


Summary  of  Selected  Raw  Data  Ratings  from  DSAT  Reliability  Study  Indicating  an 

Effect  Due  to  Experience 


SUBJ 

NUM 

EXP 

COG- 

DIFF 

MEM- 

DIFF 

CVP- 

DIFF 

EVL- 

DIFF 

DVP- 

DIFF 

DVP- 

FREQ 

DVP- 

CRIT 

DSI 

1 

2 

4.1 

3.3 

3.6 

3.3 

3.3 

4.4 

4.6 

4.8 

2 

2 

3.7 

3.3 

3.3 

3.3 

3.3 

5.0 

7.0 

4.9 

5 

2 

6.5 

6.3 

6.0 

5.6 

5.6 

3.6 

6.4 

6.3 

6 

2 

6.1 

5.6 

4.2 

4.0 

0.0 

0.0 

0.0 

3.3 

9 

2 

5.7 

4.7 

3.6 

4.4 

3.9 

5.4 

5.7 

5.2 

3 

1 

5.3 

4.9 

4.6 

4.3 

6.7 

10.0 

10.0 

6.9 

4 

1 

6.2 

5.5 

5.1 

4.3 

5.6 

6.1 

7.7 

6.7 

7 

1 

8.7 

7.4 

8.8 

9.6 

6.4 

4.1 

5.6 

7.2 

8 

1 

5.0 

5.1 

4.5 

4.8 

5.8 

6.1 

8.4 

7.0 

10 

1 

9.5 

10.0 

8.3 

8.6 

8.1 

5.2 

6.0 

7.5 

LEGEND:  Abbreviation  =  Term 


SUBJ  NUM  =  Subject  Number 

EXP  =  Subject’s  Years  of  Experience 

COG-DIFF  =  Cognition  Operator  Difficulty  Rating 

MEM-DIFF  =  Memory  Operator  Difficulty  Rating 

CVP-DIFF  =  Convergent  Production  Operator  Difficulty  Rating 

EVL-DIFF  =  Evaluation  Operator  Difficulty  Rating 

DVP-DIFF  =  Divergent  Production  Operator  Difficulty  Rating 

D  VP-FREQ  =  Divergent  Production  Operator  Frequency  Rating 

DVP-CRIT  =  Divergent  Production  Operator  Criticality  Rating 

DSI  =  Domain  Suitability  Index  Score  (See  also  Equations  1  and  2.) 

NOTE:  Subjects  with  EXP  =  >  10  years  are  in  Group  1;  subjects  with  EXP  <  10 
years  are  in  Group  2. 
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TABLE  9 

Summary  of  Correlations  Between  Experience  and  the  Difficulty  Dimension,  the 
Divergent  Production  Operator,  and  DSI  Score 


COG- 

MEM- 

CVP- 

EVL- 

DVP- 

DVP- 

DVP- 

DSI 

DIFF 

DIFF 

DIFF 

DIFF 

DIFF 

FREQ 

CRIT 

Rho  -  0.537 

0.609 

0.507 

0.498 

0.715 

0.464 

0.372 

0.658 

Level  of  Significance  - 

0.109 

0.062 

0.134 

0.143 

0.020 

0.177 

0.290 

0.039 

LEGEND:  Abbreviation  =  Term 


COG-DIFF  =  Cognition  Operator  Difficulty  Rating 
MEM-DIFF  =  Memory  Operator  Difficulty  Rating 
CVP-DIFF  =  Convergent  Production  Operator  Difficulty  Rating 
EVL-DIFF  =  Evaluation  Operator  Difficulty  Rating 
DVP-DIFF  =  Divergent  Production  Operator  Difficulty  Rating 
DVP-FREQ  =  Divergent  Production  Operator  Frequency  Rating 
DVP-CRIT  =  Divergent  Production  Operator  Criticality  Rating 
DSI  =  Domain  Suitability  Index  Score  (See  also  Equations  1  and  2.) 

NOTE:  Rho  is  the  correlation  coefficient  and  Level  of  Significance  is  the  probability 
of  getting  a  higher  value  of  Rho  by  chance  alone. 


(relative  to  goal  achievement)  alternatives  for  a  given  set  of  conditions. 

Chiesi,  et  al.  determined  that  experts  also  tended  to  show  very  little  variation  in 
their  responses  to  a  given  situation.  That  is,  the  mental  models  of  the  experts  appeared 
to  be  converging.  It  is  interesting  to  note  that  the  standard  deviation  for  the  DSI  score 
of  Group  1  (expert)  subjects  was  0.305,  while  Group  2  (novice)  subjects  was  1.075, 
which  indicates  that  experts  shared  a  greater  similarity  in  their  perception  of  the 
domain  than  did  novices.  Also,  the  difference  in  mean  DSI  scores  for  thetwo  groups 
was  significantly  different  at  the  0.0025  level  (see  Table  10). 

Chi,  Glaser,  and  Rees  (1981)  also  report  similar  findings.  Experts  in  their  study 
were  able  to  give  more  accurate  estimations  of  problem  difficulty  than  novices. 

Experts  all  recognized  the  basic  underlying  concepts  of  the  problems  which  allowed  the 
more  accurate  estimates,  while  novices  could  only  focus  on  the  surface  level  attributes 
of  the  problem,  making  their  difficulty  estimates  less  accurate.  This  statement  is 
supported  by  the  higher  variation  among  novice  DSI  scores. 

There  are  several  implications  of  the  results  of  the  DSAT  reliability  study.  First,  the 
results  indicate  the  DSAT  questionnaire  is  fairly  reliable  based  on  the  examination  of 
the  measures  of  interest  presented  above.  Next,  the  sensitivity  of  the  DSAT  (even  in 
its  current,  unrefined  format)  to  differences  in  levels  of  expertise  and  concurrence  of 
these  results  with  other  research  supports  the  validity  of  the  tool.  Finally,  the  results  of 
this  study  suggest  the  DSAT  questionnaire  will  have  a  positive  role  in  future  research 
activities  related  to  cognitive  engineering  and  knowledge  acquisition  as  well  as 
assessing  domain  suitability  for  expert  systems  technology. 
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TABLE  10 


ANOVA  Summary  Table  for  Comparing  DSI  Means  Based  on  Experience  Level 


SOURCE 

DEGREES  OF 
FREEDOM 

SUM  OF 
SQUARES 

MEAN 

SQUARE 

OBSERVED 

F-STATISTIC 

EXPERIENCE 

1 

11.6640 

11.6640 

18.69 

ERROR 

8 

4.9920 

0.6240 

TOTAL 

9 

16.6560 

R  SQUARE  =  0.7003 


PR  >  F  =  0.0025 


8.0  FUTURE  RESEARCH 

There  are  still  several  areas  to  be  addressed  by  future  research  to  fully  develop  the 
DSAT.  These  include  development  of  an  empirically  established  domain  suitability 
index  (DS1),  refinement  and  validation  of  the  DSAT  as  an  operational  tool,  and  other 
application  areas  such  as  knowledge  acquisition,  definition  of  the  user’s  mental 
structuring  of  the  task,  determination  of  the  functional  allocation  of  duties  between  the 
human  and  the  system,  and  identification  of  areas  of  expert  systems  technology 
requiring  additional  research. 

8.1  DEVELOPMENT  OF  THE  DSI  ALGORITHM.  Perhaps  the  most  difficult  and 
important  step  to  achieving  the  original  goal  of  DSAT  is  the  development  of  an 
empirically  established  algorithm  for  producing  a  domain  suitability  index  score  that 
accurately  reflects  the  capabilities  and  limitations  of  expert  systems  technology  relative 
to  the  elements  specified  in  the  SOI  model.  The  DSAT  currently  provides  a  useful 
description  of  the  domain;  a  validated  algorithm  will  provide  high  confidence  in  the 
recommendations  of  the  domain  suitability  index  score. 

8.2  REFINEMENT  OF  DSAT.  Continued  refinement  of  the  DSAT  questionnaire 
and  administrative  procedures  will  increase  the  reliability  of  the  data  collected  and 
quality  of  that  data.  One  potential  area  of  research  involves  the  development  of  more 
specific  domain  examples  that  may  increase  the  domain  expert’s  ability  to  recall 
activities  in  the  domain.  Examination  of  the  expected  benefits  of  this  more  focused 
approach  balanced  against  the  costs  of  producing  a  number  of  domain-specific 
questionnaires  would  be  useful. 

A  related  area  of  research  is  the  implementation  of  an  automated  DSAT.  Using 
the  power  of  the  computer  coupled  with  a  hypertext  format,  the  DSAT  is  envisioned  as 
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moving  from  a  paper  and  pencil  questionnaire  to  a  computer-supported  aid  for 
knowledge  acquisition  and  domain  evaluation.  Supporting  the  DSAT  entirely  via  the 
interactive  computer  will  enhance  both  the  speed  of  data  collection  and  the  range  of 
domain  experts  that  can  be  interviewed  thereby  providing  better  domain  coverage. 

This  accumulation  of  data  from  different  sources  could  allow  the  knowledge  engineer 
to  develop  a  better  understanding  of  the  domain. 

Adaptation  of  the  DSAT  to  allow  comparison  of  existing  expert  systems  for  a  given 
domain  is  another  necessary  topic  for  future  research.  The  utility  of  having  a  common 
standard  of  comparison  for  two  (or  more)  expert  systems  for  a  particular  application 
appears  to  be  high.  Some  work  is  necessary  to  modify  the  data  reduction/analysis  used 
in  the  current  tool  to  account  for  the  different  expert  systems  (perhaps  different 
domain  suitability  index  weighting  factors  based  on  the  strengths  of  the  particular 
expert  system). 

As  the  DSAT  is  used  for  increasingly  diverse  domains,  data  must  be  collected  to 
establish  its  true  validity  as  a  domain  evaluation  tool.  Data  regarding  the  outcomes  of 
expert  systems  development  programs  and  the  initial  recommendations  of  DSAT  can 
be  used  to  establish  this  validity.  Also,  subjective  data  from  both  domain  experts  and 
knowledge  engineers  can  be  collected  to  refine  the  DSAT  and  document  its 
operational  utility. 

8.3  OTHER  APPLICATIONS  OF  DSAT.  There  are  several  possible  applications  of 
the  domain  definition  capabilities  of  the  DSAT  methodology.  Future  research  should 
address  the  role  of  DSAT  in  knowledge  acquisition,  cognitive  engineering,  functional 
allocation,  and  identification  of  expert  systems  technology  areas  requiring  additional 
research. 

8.3.1  KNOWLEDGE  ACQUISITION.  A  possible  role  f^r  DSAT  may  be  in  the 
area  of  knowledge  acquisition.  The  domain  representation  gleaned  from  a  DSAT 
analysis  could  be  a  useful  aid  in  focusing  the  activities  of  a  knowledge  engineer  during 


the  knowledge  elicitation  process.  The  Level  2  display  of  the  gifted  child  assessment 
domain,  (which  showed  cognition  and  evaluation  as  leading  operators)  for  example 
would  suggest  questions  regarding  how  the  specialist  discovers  and  recognizes  traits  in 
an  individual  and  then  how  those  traits  are  evaluated. 

The  benefit  to  knowledge  engineers  is  that  with  comparatively  small  investment  of 
resources,  the  DSAT  provides  a  fast  method  of  determining  what  areas  of  the  domain 
are  critical  and  difficult  in  the  domain.  This  knowledge  is  particularly  useful  when  the 
knowledge  engineer  is  pursuing  a  domain  with  which  he  is  unfamiliar.  Taking  a 
rudimentary  domain  understanding.  DSAT  is  able  to  highlight  the  areas  of  greatest 
domain  importance,  allowing  the  knowledge  engineer  to  proceed  immediately  to  the 
"high  payoff  aspects  of  the  domain.  This  can  save  time  and  money  in  the  development 
of  the  system  by  reducing  the  impact  of  the  "knowledge  acquisition  bottleneck." 

8.3.2  COGNITIVE  ENGINEERING.  The  field  of  cognitive  engineering  (see 
Woods  and  Roth,  1988)  which  seeks  to  develop  a  principle-driven,  applied  cognitive 
science  based  on  the  fundamentals  and  theories  of  cognitive  psychology  and  other 
related  fields,  could  also  benefit  from  the  DSAT  methodology. 

A  key  challenge  for  cognitive  engineering  is  developing  a  method  of  identifying  the 
foundations  of  expert  task  performance  (Gitomer  and  Glaser,  1987).  The  studies  of 
Chi,  et  al.  (1981)  and  others  have  clearly  shown  that  experts  perform  at  high  levels  due 
to  different  methods  of  organizing  specific  elements  of  domain  knowledge  (Roth  and 
Woods,  1988).  The  supporting  results  obtained  during  the  DSAT  reliability  analysis 
indicate  the  potential  usefulness  of  DSAT  for  describing  how  an  expert  structures 
domain-specific  knowledge. 

Based  on  the  potentially  high  contribution  the  DSAT  methodology  can  offer  in 
describing  the  structure  of  domain-specific  knowledge,  further  studies  investigating  the 
utility  of  DSAT  for  these  and  other  cognitive  engineering  and  cognitive  science  related 


research  should  be  pursued. 
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8.3.2  FUNCTIONAL  ALLOCATION.  By  specifying  the  relevant  operations  and 
information  contents  of  the  domain,  DSAT  allows  a  rapid  and  accurate  comparison  of 
both  system  strengths  and  weaknesses,  as  well  as  user  strengths  and  limitations.  With 
an  understanding  of  the  essential  domain  information  attributes.  Cue  entity  (man  or 
machine)  best  equipped  to  support  and  deal  with  that  information  element  can  be 
assigned  responsibility  for  that  element  in  the  overall  system  design.  The  resulting 
allocation  of  functions  (tasks)  between  user  and  (expert)  system  will  result  in  the 
greatest  level  of  overall  system  effectiveness  and  performance. 

8.3.3  IDENTIFICATION  OF  EXPERT  SYSTEMS  RESEARCH  TOPICS.  Finally, 
domains  that  score  high  on  the  DSI  scale  can  be  examined  to  determine  what  domain 
information  elements  make  them  unsuitable  candidates  for  expert  systems  technology 
solutions.  The  common  traits  of  these  domains  can  be  specified  to  define  unsupported 
activities.  Expert  system  designers,  cognitive  engineers,  software  designers,  human 
factors  engineers,  and  artificial  intelligence  scientists  can  then  investigate  alternative 
means  for  developing  technology  to  implement  these  activities  in  future  expert  systems. 


9.0  CONCLUSIONS 


The  are  several  important  general  conclusions  to  be  drawn  from  this  research. 
First,  the  field  of  expert  systems  can  benefit  from  the  inclusion  of  basic  principles  of 
human  cognition  in  the  selection  of  potential  application  domains.  Second, 
development  and  application  of  a  principle  driven  design  approach  for  building  expert 
systems  can  not  only  reduce  the  resources  required  to  achieve  a  working  system,  but 
also  will  ultimately  result  in  a  better  accepted,  more  capable  system  that  meets  the 
performance  objectives  of  the  project.  Third,  theories  of  human  intelligence  can  be 
successfully  applied  to  varying  domains  to  define  the  information  requirements  of 
those  domains  and  provide  decision-makers  with  useful  knowledge  of  the  domain’s 
suitability  for  an  expert  systems  technology  solution. 

There  are  also  several  important  specific  conclusions  to  be  drawn  from  this 
research.  The  working  hypotheses  of  this  research  are  (see  Section  6.1): 

Hypothesis  1: 

The  DSAT  will  provide  a  description  of  the  information  types  and  processes  used  for 
performance  of  domain  tasks  by  the  human  expert. 

Hypothesis  2: 

The  DSAT  will  be  easily  administered,  analyzed ,  and  interpreted. 

Hypothesis  3: 

The  DSAT  will  possess  a  high  degree  of  reliability  for  describing  domain  information 
requirements. 

Hypothesis  4: 

The  DSAT  will  possess  a  high  degree  of  validity  for  describing  domain  information 
requirements. 

The  DSAT  implementation  and  results  described  in  Section  7.1  suggest  the  DSAT 

methodology  is  capable  of  obtaining  domain  information  and  discriminating  between 
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different  domains.  This  indicates  support  for  accepting  Hypothesis  I,  that  the  DSAT  is 
useful  for  examining  human  task  domains. 

Acceptance  of  Hypothesis  2  is  also  supported  since  these  results  suggest  the  DSAT 
methodology  gathers  domain  information  quickly  and  presents  domain  data  in  a 
readily  understandable  format,  lending  itself  to  ease  of  application  by  practitioners. 

The  results  described  in  Section  7.3  indicate  the  DSAT  (in  its  current  form)  is  fairly 
reliable  for  most  domain  information  elements,  suggesting  support  for  accepting 
Hypothesis  3.  Additional  refinement  of  the  tool  is  needed  to  improve  the  reliability  of 
the  divergent  production  operator  and  difficulty  dimension  ratings. 

While  not  formally  addressed  in  the  experimental  design  of  this  research,  some 
support  of  acceptance  for  Hypothesis  4  was  also  determined.  Based  on  the  similar 
findings  of  other  researchers  identified  in  Section  7.3.5.,  some  concurrent  validity  for 
the  DSAT  can  be  inferred.  Also,  by  virtue  of  using  an  established  theory,  the 
methodology  has  a  high  degree  of  construct  validity. 

While  the  Domain  Suitability  Assessment  Tool’s  goal  of  a  complete  domain 
assessment  methodology  has  not  yet  been  totally  achieved,  the  critical  first  steps  have 
been  successfully  taken.  A  methodology,  with  its  foundation  in  an  established  theory  of 
cognition,  has  been  developed  and  demonstrated  as  a  reliable  means  of  defining  the 
information  requirements  of  a  domain.  It  is  now  possible  to  pursue  development  of  an 
accurate  translation  of  those  domain  information  requirements  into  meaningful 
estimates  of  the  degree  of  support  provided  by  current  expert  systems  technology. 
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APPENDIX  A 

DOMAIN  SUITABILITY  ANALYSIS  TOOL  ( DS AT ) 
INSTRUCTIONS  AND  QUESTIONNAIRES 

Part  I.  ASSESSMENT  OF  DOMAIN  STRUCTURAL  COMPONENTS 

The  following  5  sections  of  6  questions  examine  the 
information  content  of  the  domain  under  consideration.  First, 
examine  the  section  title  and  accompanying  description.  If 
the  type  of  information  described  is  evident  in  the  domain, 
proceed  to  each  question  in  that  section,  answering  "yes"  or 
"no."  Answer  "yes"  if  the  example  of  the  information  element 
occurs  in  the  domain,  and  "no"  if  it  does  not.  If  you  are 
unsure  about  an  item,  answer  "yes;"  you  will  be  able  to 
examine  it  more  thoroughly  in  Part  II.  After  you  have 
examined  all  5  sections,  and  answered  the  applicable 
questions,  proceed  to  Part  II. 
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SECTION  1.  VISUAL  CONTENTS.  To  perform  successfully  in  this 
domain,  is  it  necessary  to  see  and  process  items  of  a  visual 
nature,  such  as  trees,  parts,  faces,  buildings,  or  other 
objects  that  can  be  immediately  perceived  or  comprehended  as 
visual  forms?  For  example,  in  the  domain  of  diagnostic 
medicine,  a  doctor  must  be  able  to  process  information  such 
as  the  physical  symptoms  (swelling,  discoloration,  skin  rash, 
etc.),  and  general  condition  through  primarily  visual  means. 

YES _  (answer  remaining  questions)  NO _  (proceed  to 

next  section) 

YES  -  NO 

_  -  _  A.  VISUAL  UNITS.  Any  single  item  with  a  unique 

identity.  Examples:  a  word,  a  silhouette,  a 
universal  product  code  symbol,  a  dot,  an 
aircraft. 

_  -  _  B.  VISUAL  CLASSES.  Classification  of  images 

into  related  groups.  Examples:  red  items, 
biplanes,  cubist  art,  deciduous  trees,  printer 
fonts . 

_  -  _  C.  VISUAL  RELATIONS.  A  logical  connection 

between  two  images.  Examples:  one  girl  taller 
than  another,  similarities  of  Porsche  and 
Corvette  body  styles. 

_  -  _  D.  VISUAL  SYSTEMS.  Three  or  more  items 

composing  a  collective  whole.  Examples:  the 
placement  of  22  football  players  before  a  play, 
the  positions  of  all  aircraft  waiting  in  a 
holding  pattern. 

_  -  _  E.  VISUAL  TRANSFORMATIONS.  Modification  of  an 

existing  image  into  another  form. 

Examples:  visualizing  the  rotation  of  an  object 
by  270  degrees,  imagining  how  a  landscape  seen 
in  the  summer  appears  in  the  winter. 

_  -  _  F.  VISUAL  IMPLICATIONS.  Foreseeable 

consequences  evidenced  by  the  current  visual 
state.  Examples:  seeing  a  brown  apple  and 
knowing  not  to  eat  it,  seeing  a  red  light  and 
stopping. 


80 


SECTION  2.  AUDITORY  CONTENTS.  Does  successful  performance 
depend  on  the  ability  to  perceive  or  comprehend  information 
of  an  auditory  nature,  such  as  verbal  commands,  warning 
tones,  music,  or  interpretive  noises  (like  the  hum  of  a 
well-tuned  engine,  the  squeal  of  a  loose  fan-belt,  or  other 
sounds  indicating  things  aie  "right"  or  "wrong")?  In  the 
diagnostic  medicine  example,  the  doctor  must  hear  the 
patient's  heartbeat,  air  filling  the  lungs,  and  the  patient's 
own  comments  to  arrive  at  an  accurate  diagnosis. 

YES _  (answer  remaining  questions)  NO _  (proceed  to 

next  section) 

YES  -  NO 

-  _  A.  AUDITORY  UNITS.  A  single  sound  with  a 

unique  identity.  Examples:  a  musical  note,  a 
spoken  word,  a  dog's  bark,  a  car  horn. 

_  -  _  B.  AUDITORY  CLASSES.  Classification  of  sounds 

into  groups.  Examples:  words  with  the  "o o" 
sound,  loud  sounds,  ragtime  tunes,  rock  and  roll 
music. 

_  -  _  C.  AUDITORY  RELATIONS.  Logical  connections 

between  two  sounds.  Examples:  two  tones  an 
octave  apart,  a  waltz  compared  to  a  march,  a 
cannon  being  louder  than  a  rifle. 

_  -  _  D.  AUDITORY  SYSTEMS.  Three  or  more  sounds 

integrated  to  form  a  unique  whole.  Examples:  a 
melody,  a  rhythm,  the  phonetic  alphabet  (ALPHA, 
BRAVO,  CHARLIE,  DELTA,  etc.). 

_  -  _  E.  AUDITORY  TRANSFORMATIONS.  Modification  of 

an  existing  sound  or  pattern.  Examples:  a  jazz 
solo  based  on  the  original  melody,  disguising 
one's  voice. 

_  -  _  F.  AUDITORY  IMPLICATIONS.  Information 

suggested  by  a  given  sound.  Examples:  hearing  a 
bell  and  knowing  class  is  over,  hearing  brakes 
screech  and  bracing  for  impact. 
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SECTION  3.  SYMBOLIC  CONTENTS.  Does  the  human  operator  need 
to  understand  or  process  information  of  a  symbolic  nature? 
Symbols,  such  as  a  letter,  number,  musical  note,  or  a  **$"  are 
arbitrary  items  that  have  no  intrinsic  significance,  until 
placed  in  a  specific  context.  The 

doctor  consults  a  variety  of  charts,  instruments,  and  other 
symbolic  representations  of  the  patient's  state  when 
performing  diagnosis  of  the  patient's  symptoms. 

YES _  (answer  remaining  questions)  NO _  (proceed  to 

next  section) 

YES  -  NO 

_  -  _  A.  SYMBOLIC  UNITS.  A  single  item  with  a  unique 

identity.  Examples:  a  plus  (+)  sign,  a  tank 
silhouette,  a  square,  a  Cyrillic  letter,  the 
Statue  of  Liberty. 

_  -  _  B.  SYMBOLIC  CLASSES.  Classification  of  symbols 

into  groups.  Examples:  mathematical  symbols, 
Arabic  numerals,  military  rank  insignia, 
national  flags. 

_  -  _  C.  SYMBOLIC  RELATIONS.  Logical  connections 

between  two  symbols.  Examples:  "6"  is  greater 
than  "5,"  a  silver  star  (general)  outranks  a 
silver  eagle  (colonel). 

_  -  _  D.  SYMBOLIC  SYSTEMS.  Three  or  more  symbols 

composing  a  collective  whole.  Examples: 
telephone  numbers,  words,  stop  lights  (red  * 
stop,  yellow  -  caution,  green  -  go) . 

_  -  _  E.  SYMBOLIC  TRANSFORMATIONS.  Modification  of 

an  existing  symbol  or  set  of  symbols.  Example: 
rearranging  scrambled  letters  into  words, 
converting  written  words  into  their  audible 
counterparts  when  reading  aloud. 

_  -  _  F.  SYMBOLIC  IMPLICATIONS.  Information 

suggested  by  a  given  symbol.  Examples:  if  a 
given  symbol  is  a  friendly  or  enemy  aircraft,  if 
ADV  stands  for  "advance"  or  "adversary,"  if  "*" 
means  multiply  or  divide. 
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SECTION  4.  SEMANTIC  CONTENTS.  Is  it  necessary  to  maintain  a 
repertoire  of  words  and  ideas  that  carry  an  abstract  meaning 
that  are  immediately  comprehended  when  the  stimulus  is 
perceived?  Semantic  contents  pertains  to  information  in  the 
form  of  an  idea  or  mental  image.  Meaningful  pictures,  words, 
and  verbal  communication  are  examples  of  this  concept.  For 
example,  a  patient  relates  he  has  a  "burring  sensation"  in 
his  lower-right  portion  of  his  abdomen.  To  the  doctor,  these 
words  strongly  suggest  appendicitis. 

YES _  (answer  remaining  questions)  NO _  (proceed  to 

next  section) 

YES  -  NO 

_  -  _  A.  SEMANTIC  UNITS.  Single  ideas  or  concepts. 

Examples:  the  meanings  of  words,  the  sound  of  a 
particular  musical  note,  a  baseball  player's 
batting  average. 

_  -  _  B.  SEMANTIC  CLASSES.  Groupings  of  similar 

concepts.  Examples:  kinematic  equations  of 
motion,  musical  scales,  farm  animals,  types  of 
viruses,  styles  of  architecture. 

_  -  _  C.  SEMANTIC  RELATIONS.  Connections  logically 

made  between  two  concepts.  Examples:  tie 
between  bond  prices  and  interest  rates, 
correlation  of  height  and  weight. 

_  -  _  D.  SEMANTIC  SYSTEMS.  Three  or  more  integrated 

concepts  composing  a  complex  whole.  Examples:  a 
story,  the  rules  of  tennis,  objects  on  a  radar 
screen,  English  language. 

_  -  _  E.  SEMANTIC  TRANSFORMATIONS.  Potential  changes 

of  interpretations  of  objects  and  situations. 
Examples:  design  changes  needed  to  meet  new 
specifications,  a  court  decision  based  on  new 
evidence. 

_  -  _  F.  SEMANTIC  IMPLICATIONS.  Consequences  or 

outcomes  suggested  by  specific  concepts. 
Examples:  repercussions  of  unilateral 
disarmament,  fiscal  policy  decisions  based  on 
the  leading  economic  indicators. 
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SECTION  5.  BEHAVIORAL  CONTENTS.  To  perform  successfully  in 
this  domain,  is  the  perception  and  understanding  of  human 
attitudes,  needs,  moods,  ideas,  and  intentions  required? 

These  information  elements  are  primarily  evident  in  domains 
involving  human  interaction,  but  can  be  required  for 
individual  activities.  To  illustrate,  our  diagnostic 
physician  will  attempt  to  gauge  the  patient's  physical  state 
by  noting  the  patient's  alertness,  tone  of  voice,  moodiness, 
and  other  exhibited  behaviors. 

YES _  (answer  remaining  questions)  NO _  (proceed  to 

Part  II) 

YES  -  NO 

_  -  _  A.  BEHAVIORAL  UNITS.  Single  elements  of 

expression.  Examples:  a  frown,  a  wink, 
laughter,  posture,  gestures,  tone  of  voice. 

_  -  _  B.  BEHAVIORAL  CLASSES.  Groupings  of  similar 

expressions.  Examples:  happy  faces,  depressed 
body  postures,  aggressive  gestures. 

_  -  _  C.  BEHAVIORAL  RELATIONS.  Connections  between 

behaviors  based  upon  social  understanding. 
Examples:  waving  when  you  say  good-bye,  shaking 
hands  when  you  are  introduced. 

_  -  _  D.  BEHAVIORAL  SYSTEMS.  Relevant  behaviors 

comprising  a  social  situation.  Examples: 
social  etiquette,  acceptable  conduct  on  the 
football  field,  Washington  "VIP"  protocol. 

_  -  _  E.  BEHAVIORAL  TRANSFORMATIONS.  Changes  in  the 

behavioral  significance  of  a  given  situation. 
Example:  negative  stereotypes  held  about  a 
racial  group  changing  to  positive  feelings  after 
interacting  with  members  of  the  group. 

_  -  _  F.  BEHAVIORAL  IMPLICATIONS.  Foreseeable 

consequences  of  a  given  set  of  social  behaviors. 
Examples:  kicking  dirt  on  an  umpire's  shoes 
results  in  ejection  from  th°  game,  constantly 
belittling  an  employee  resv I ts  in  building 
resentment  and  eventually  th  mployee  quitting. 
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Part  II.  ASSESSMENT  OF  OPERATIONAL  COMPONENTS 

Based  upon  the  answers  provided  in  Part  I ,  Part  II 
seeks  to  identify  the  mental  operations  that  occur  in 
processing  the  relevant  information  elements  of  the  domain. 

In  this  part,  you  will  be  asked  to  indicate  which  operations 
are  utilized  by  selecting  an  example  which  most  closely 
resembles  activities  in  your  domain.  After  selecting  the 
proper  example,  please  indicate  your  selection  by  providing  a 
rating  of  the  frequency  of  occurrence  and  criticality  of  the 
element  to  satisfactory  performance  in  the  domain  in  the 
space  beside  the  appropriate  example.  If  two  (or  all) 
examples  of  a  specific  domain  element  occur  in  your  domain, 
please  rate  only  the  example  with  the  highest  number.  For 
example,  if  all  three  examples  applied,  only  example  number 
three  (3)  would  be  rated;  if  examples  one  (1)  and  two  (2) 
applied,  only  two  would  be  rated.  If  none  of  the  examples 
are  appropriate  to  your  domain,  leave  the  question  blank. 
Indicate  your  rating  for  each  element  using  the  scale  shown 
below: 
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| 

4 
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1  1  ‘ 
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1  2 
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| 
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| 

CRITICAL 

SECTION  1.  COGNITION.  This  operation  involves  the 
discovery,  comprehension,  awareness,  or  recognition  of 
information  elements.  In  its  broadest  sense,  understanding . 
More  specifically,  the  process  of  structuring  items  of 
information.  Comprehending  the  blob  of  color  is  a  garbage 
truck,  understanding  the  meaning  of  words  organized  as  a 
poem,  adding  a  column  of  numbers,  and  interpreting  the  body 
language  of  an  angry  customer  are  all  examples  of  cognition. 

YES _  (answer  remaining  questions)  N0_ _  (proceed  to 

next  section) 

F  -  C 

A.  COGNITION  OF  VISUAL  UNITS: 

_  -  _  1.  Recognizing  a  simple,  distinct,  visual  form. 

Seeing  four  connected  linos  au  a  square, 
identifying  a  color  as  "rod." 

_  -  _  2.  Recognizing  a  moderately  complex  visual  form 

along  gross  dimensions.  Identifying  an  object 
as  a  building  or  recognizing  a  car  versus  a  jeep 
or  a  tank. 

_  -  _  3.  Recognizing  a  complex  visual  form  in  varying 

degrees  of  background  clutter.  Resolving  an 
infra-red  sensor  image  as  an  enemy  tank  among 
other  heat  generating  sources  or  finding  a 
particular  face  in  a  crowd. 

F  -  C 

B.  COGNITION  OF  VISUAL  CLASSES: 

_  -  _  1.  Classifying  visual  items  by  a  few  simple  or 

readily  apparent  attributes.  Given  a  set  of 
objects,  which  items  are  blue,  which  items  have 
wheels,  which  are  biplanes. 

_  -  _  2.  Classifying  visual  forms  on  several  moderately 

ambiguous  or  not  readily  apparent  dimensions. 
Classifying  an  item  as  an  oak,  sycamore,  or  elm 
leaf  or  classifying  a  building  as  classic, 
modem,  or  futurist  architecture. 

_  -  _  3.  Classifying  complex  visual  forms  on  ambiguous  or 

esoteric  factors.  Defining  a  painting  as  a  late 
impressionist  period  piece  or  classifying  a 
radar  blip  as  a  fighter  plane  or  a  passenger 
transport. 

F  -  C 

C.  COGNITION  OF  VISUAL  RELATIONS: 

_  -  _  1.  Comprehending  simple  relationships  among  a  few 

distinct  visual  forms.  Understanding  apples  and 
stop  signs  are  "red"  or  that  a  book  is  "on"  a 
table. 

_  -  _  2.  Comprehending  more  involved  relationships  among 

a  few  visual  forms.  Producing  a  polygon  to 
complete  a  sequence  such  as  specifying  "hexagon" 
to  complete  the  sequence  of 
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triangle,  square,  pentagon,  or  a  book  may  still 
be  "on”  the  table,  even  though  it  has  been 
placed  "behind"  a  box  also  on  the  table. 

3.  Comprehending  complex  relationships  among 
potentially  ambiguous  visual  forms  and 
dimensions.  Understanding  how  a  Porsche  and  a 
Corvette  are  related  or  the  visual  relation  of 
an  F-?.6  and  F-15  aircraft. 

D.  COGNITION  OF  VISUAL  SYSTEMS: 

1.  Perceiving  the  static  arrangement  and  position  a 
few  simple  visual  forms.  The  positions  of 
appliances  in  a  kitchen  or  the  position  of  an 
object  in  a  two-dimensional  plane. 

2.  Perceiving  the  arrangement  and  spatial  position 
of  several  moderately  complex  objects  that  can 
move  predictably.  Identifying  a  football 
formation  based  on  the  positions  of  the  players 
or  perceiving  the  position  of  a  single  aircraft 
given  its  flight  plan. 

3.  Perceiving  the  arrangement  and  spatial  positions 
of  members  of  a  complex  system  of  visual  forms 
in  varying  degrees  of  background  clutter. 
Understanding  the  positions  of  all  aircraft  in  a 
holding  pattern  over  a  busy  airport  or  using 
terrain  feature  information  to  guide  a  missile 
to  its  target. 

E.  COGNITION  OF  VISUAL  TRANSFORMATIONS: 

1.  Understanding  the  impact  of  changes  on  a  simple 
visual  object  after  some  specific,  significant 
changes  occur.  Recognizing  a  block  has  rotated 
270  degrees  or  removing  a  side  from  a  square  to 
form  a  triangle. 

2.  Understanding  the  result  after  several  changes 
are  imposed  on  a  moderately  complex  visual 
scene.  Selecting  the  correct  piece  to  complete 
a  puzzle  or  indicating  the  arrangement  of 
appliances  in  a  kitchen  has  been  changed. 

3.  Understanding  the  impact  of  subtle  changes  in  a 
complex  visual  form.  Recognizing  a  man's  face 
after  aging  30  years  or  perceiving  how  design 
changes  will  affect  the  visual  appearance  of  a 
vehicle. 

F.  COGNITION  OF  VISUAL  IMPLICATIONS: 

1.  Recognizing  the  obvious  consequences  inherent  in 
an  unambiguous  visual  scene.  Planning  a  simple 
circuit  diagram  or  navigating  a  simple  maze. 

2.  Recognizing  the  consequences  inherent  in  a 
possibly  ambiguous  visual  scene.  Planning  a 
route  to  get  from  Boston  to  Long  Island  or 
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determining  if  a  piece  of  lumber  meets  minimum 
standards  for  grain  and  knots. 

3 .  Understanding  the  possible  consequences  in  an 
ambiguous  visual  scene.  Planning  an 
airborne-intercept  route  based  on  the  opponent's 
maneuvers  or  docking  a  spacecraft  with  another 
moving  craft. 

G.  COGNITION  OF  AUDITORY  UNITS: 

1.  Recognizing  a  simple,  distinct,  auditory 
stimulus.  Hearing  a  school  bell  or  recognizing  a 
car  horn. 

2.  Recognizing  a  moderately  complex  auditory 
stimulus  along  gross  dimensions.  Identifying  a 
specific  tone  from  several  candidate  tones  or 
recognizing  examples  of  rock  music. 

3.  Recognizing  a  complex  auditory  stimulus  in 
varying  degrees  of  background  clutter. 
Picking-out  a  specific  word  in  a  string  of 
speech  or  identifying  an  out-of-tune  violin  in 
an  orchestra. 

H.  COGNITION  OF  AUDITORY  CLASSES: 

I.  Classifying  auditory  stimuli  by  a  few  simple  or 
readily  apparent  attributes,  such  as  tones  less 
than  5,000  Hz  or  sounds  between  10  and  10,000 
decibels. 

2.  Classifying  auditory  stimuli  on  several 
moderately  ambiguous  or  not  readily  apparent 
dimensions.  Classifying  a  dog's  bark  as  loud  or 
soft  or  identifying  a  car  horn  as  either  high  or 
low-pitched. 

3.  Classifying  complex  auditory  stimuli  on 
ambiguous  or  esoteric  factors.  Describing  a 
musical  piece  as  being  composed  by  Chopin  or 
classifying  a  submarine  from  the  sound  of  its 
screws  under  water. 

I.  COGNITION  OF  AUDITORY  RELATIONS: 

1.  Comprehending  simple  relationships  among  a  few 
distinct  auditory  stimuli.  Understanding  that 
"middle  C"  is  lower  than  "high  C"  or  that  one 
tone  is  in  the  same  key  as  a  given  tone. 

2 .  Comprehending  more  involved  relationships  among 
a  few  auditory  stimuli.  Understanding  the 
relationships  among  notes  in  a  chromatic  scale. 

3.  Comprehending  complex  relationships  among 
potentially  ambiguous  auditory  stimuli  and 
dimensions.  Understanding  how  a  waltz  and  a 
march  are  related  or  the  similarity  of  men's 
voices. 
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J.  COGNITION  OF  AUDITORY  SYSTEMS: 

1.  Perceiving  the  integration  of  a  few  simple 
sounds  to  form  a  unique  whole.  Understanding  a 
bugler's  signal  to  charge  or  a  simple  drum 
cadence. 

2.  Perceiving  the  integration  of  several  moderately 
complex  sounds.  Identifying  some  of  the 
principle  sounds  in  a  busy  bus  station  or 
interpreting  an  expected  message  on  the 
telephone . 

3.  Perceiving  the  arrangement  and  position  of 
various  stimuli  in  a  complex  auditory  system 
with  varying  degrees  of  background  clutter. 
Understanding  a  radio  communication  that  is 
partially  jammed  or  identifying  the  position  of 
a  submarine  from  several  auditory  inputs. 

K.  COGNITION  OF  AUDITORY  TRANSFORMATIONS: 

1.  Understanding  the  impact  of  changes  on  a  simple 
auditory  stimulus  after  some  specific  changes 
occur.  Recognizing  a  tone  has  increased  in  pitch 
or  that  a  previously  steady  tone  is  now 
wavering. 

2 .  Understanding  the  result  of  several  changes 
imposed  on  a  moderately  complex  auditory 
arrangement.  Determining  the  impact  of  changing 
the  key  of  a  musical  selection  or  the  effect  of 
increasing  the  tempo  of  spoken  words  on 
intelligibility. 

3.  Understanding  the  impact  of  subtle  changes  in  a 
complex  auditory  system.  Recognizing  verbal 
commands  made  by  a  pilot  under  high-g  stress  or 
interpreting  words  that  are  sung  rather  than 
spoken. 

L.  COGNITION  OF  AUDITORY  IMPLICATIONS: 

1.  Recognizing  the  obvious  consequences  inherent  in 
an  unambiguous  auditory  stimulus.  Fastening  a 
safety  belt  when  the  car  "seat-belt  buzzer" 
sounds  or  that  a  ringing  alarm  clock  signals 
time  to  get-up. 

2.  Recognizing  the  inherent  consequences  in  a 
possibly  ambiguous  auditory  stimulus.  Knowing 
what  to  do  when  the  words  "Look  out!"  are  heard 
while  walking  through  a  hard-hat  area  or  hearing 
a  wheezing  asthma  patient  and  understanding  that 
an  attack  may  be  imminent. 

3.  Understanding  the  possible  consequences  in  an 
ambiguous  auditory  system.  Understanding  what 
is  making  a  baby  cry  or  whether  or  not  a  given 
musical  piece  is  "good." 
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M.  COGNITION  OF  SYMBOLIC  UNITS: 

1.  Recognizing  a  simple,  distinct,  symbol. 
Recognizing  a  plus  (+)  sign  or  an  ASCII 
character. 

2.  Recognizing  a  moderately  complex  symbol  along 
gross  dimensions.  Identifying  a  symbol  as  a 
tank  or  recognizing  a  company  logo. 

3.  Recognizing  a  complex  symbol  in  varying  degrees 
of  background  clutter.  Resolving  handwritten 
words  or  identifying  a  specific  Chinese 
pictograph. 

N.  COGNITION  OF  SYMBOLIC  CLASSES : 

1.  Classifying  symbols  by  a  few  simple  or  readily 
apparent  attributes.  Grouping  mathematical 
symbols  or  Arabic  numerals. 

2.  Classifying  symbols  on  several  moderately 
ambiguous  or  not  readily  apparent  dimensions. 
Recognizing  military  rank  insignia  or  correctly 
spelled  words. 

3.  Classifying  complex  symbols  on  ambiguous  or 
esoteric  factors.  Identifying  patriotic  symbols 
or  specifying  symbols  of  cultural  significance. 

O.  COGNITION  OF  SYMBOLIC  RELATIONS: 

1.  Comprehending  simple  relationships  among  a  few 
distinct  symbols.  Understanding  the 
relationship  of  "l"  and  "0"  bits  in  a  binary 
code. 

2.  Comprehending  more  involved  relationships  among 
a  few  symbols.  Understanding  that  "6"  is 
greater  than  "5" 

or  "x"  operations  are  performed  before 
operations. 

3.  Comprehending  complex  relationships  among  a  few 
symbols.  Understanding  a  silver  eagle 
(symbolizing  the  rank  of  colonel)  outranks  a 
silver  bar  (symbolizing  a  first  lieutenant) . 

P.  COGNITION  OF  SYMBOLIC  SYSTEMS: 

1.  Perceiving  three  or  more  simple  symbols 
composing  a  collective  whole.  Examples  include 
telephone  or  Social  Security  numbers  or  letters 
forming  words. 

2.  Perceiving  the  integration  of  many  moderately 
complex  symbols.  Understanding  a  code  or 
symbols  used  in  a  military  information 
distribution  system. 

3.  Perceiving  the  integration  of  many  complex 
symbols  in  different  formats.  Understanding 
words  organized  in  sentences  or  paragraphs. 
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Q.  COGNITION  OF  SYMBOLIC  TRANSFORMATIONS: 

1.  Understanding  the  impact  of  changes  on  simple 
symbols  after  some  specific  changes  occur. 
Updating  symbols  on  a  map  display  as  information 
of  their  disposition  is  obtained  or  translating 
items  from  hexadecimal  to  binary  code. 

2.  Understanding  the  impact  of  several  changes 
imposed  on  a  moderately  complex  symbol  set. 
Rearranging  scrambled  letters  to  form  a 
meaningful  word  or  the  impact  of  note  position 
in  the  bass  clef  versus  the  treble  clef. 

3 .  Understanding  the  impact  of  subtle  changes  in  a 
complex  symbol  set.  Converting  written  words 
into  their  audible  counterparts  while  reading 
aloud  or  translating  from  Russian  to  English. 

R.  COGNITION  OF  SYMBOLIC  IMPLICATIONS: 

1.  Recognizing  the  obvious  information  inherent  in 
an  unambiguous  symbol.  A  green  light  on  a 
traffic  light  implies  "go"  or  "="  implies  two 
quantities  are  equal. 

2.  Recognizing  the  information  contained  in  a 
possibly  ambiguous  symbol.  The  implication  of 
including  the  first  three  numbers  of  a  telephone 
number  (  i.e.,  (Ill)  555-  2345)  is  that  the 
number  is  long  distance,  omitting  the  prefix 
implies  a  local  call. 

3.  Recognizing  the  potential  information  contained 
in  an  ambiguous  symbol.  Understanding  a  piece  of 
evidence  found  at  a  crime  site  may  suggest 
useful  leads  for  solving  the  crime,  even  though 
its  relevance  is  unknown  at  this  time. 

S.  COGNITION  OF  SEMANTIC  UNITS: 

1.  Recognizing  a  simple  idea  or  concept.  The  sound 
of  a  particular  note  or  the  meaning  of 
non-complex  domain  elements  such  as  "dog," 

"car,"  "bed,"  etc. 

2.  Recognizing  a  moderately  complex  idea  or 
concept.  Understanding  the  meaning  of 
less-general  terms  in  the  domain  such  as  "too 
hot,"  "sell-short,"  or  other  potentially 
ambiguous  concepts  or  jargon. 

3.  Recognizing  a  complex  idea  or  concept. 
Understanding  meanings  of  abstract  terms  like 
"love,"  "beautiful,"  or  other  highly  subjective 
concepts . 

T.  COGNITION  OF  SEMANTIC  CLASSES: 

1.  Classifying  similar  concepts  by  a  few  simple 

or  readily  apparent  attributes.  Alphabetizing  a 
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list  of  words  or  identifying  a  given  formula  as 
a  kinematic  equation  of  motion. 

2.  Classifying  concepts  on  several  moderately 
ambiguous  or  not  readily  apparent  dimensions. 
Classifying  styles  of  architectural  design 
concepts  or  a  particular  action  belonging  to  a 
specific  type  of  military  maneuver. 

3.  Classifying  complex  concepts  on  ambiguous  or 
esoteric  factors.  Given  a  specific  example, 
classify  a  behavior  as  ethical  or  non-ethical. 

U.  COGNITION  OF  SEMANTIC  RELATIONS: 

1.  Comprehending  simple,  concrete  relationships 
among  a  few  concepts  or  situations. 

Understanding  the  link  between  "off"  and  "on." 

2 .  Comprehending  less  certain  relationships  among  a 
few  concepts.  Understanding  the  general 
positive  correlation  of  height  and  weight  or  the 
general  inverse  relationship  of  bond  prices  and 
interest  rates. 

3 .  Comprehending  complex  relationships  among 
potentially  ambiguous  concepts  or  ideas. 
Understanding  the  relationship  of  the  concept  of 
"patriot"  to  the  concept  of  "nationalism." 

V.  COGNITION  OF  SEMANTIC  SYSTEMS: 

1.  Perceiving  the  integration  of  three  or  more 
simple,  unambiguous  concepts  which  comprise  a 
unique  whole.  Understanding  the  rules  of  tennis 
or  basic  mathematics. 

2 .  Perceiving  the  integration  of  several  moderately 
complex  concepts  in  potentially  ambiguous 
contexts.  Identifying  key  themes  in  a  newspaper 
article. 

3.  Perceiving  the  integration  of  many  complex 
concepts  in  generally  ambiguous  contexts. 
Understanding  the  central  themes  of  "Mein  Kampf" 
or  comprehending  the  meaning  of  a  clergyman's 
sermon. 

W.  COGNITION  OF  SEMANTIC  TRANSFORMATIONS: 

1.  Understanding  the  impact  of  a  few  specific 
changes  imposed  on  a  simple  conceptual  system. 
Adding  water  to  a  glass  changes  its  meaning  from 
"empty"  to  "full." 

2 .  Understanding  the  impact  of  several  changes 
imposed  on  a  moderately  complex  conceptual 
system.  Revising  a  report  for  a  different 
audience  than  originally  intended. 

3.  Understanding  the  impact  of  subtle  or  major 
changes  in  a  complex  conceptual  system  or 
situation.  Understanding  the  design  changes 


needed  to  implement  new  system  requirements  or 
reinterpretation  of  a  court  decision  based  upon 
new  evidence. 

X.  COGNITION  OF  SEMANTIC  IMPLICATIONS: 

1.  Recognizing  the  obvious  consequences  inherent  in 
a  unambiguous  concept  or  situation.  Seeing  a 
simple  circuit  diagram  or  navigating  a  simple 
maze. 

2.  Recognizing  the  consequences  inherent  in  a 
possibly  ambiguous  visual  scene.  Planning  a 
route  to  get  from  Boston  to  Long  Island  or 
determining  if  a  piece  of  lumber  meets  minimum 
standards  for  grain  and  knots. 

3.  Understanding  the  possible  consequences  in  an 
ambiguous  visual  scene.  Planning  an 
airborne- intercept  route  based  on  the  opponent's 
maneuvers  or  docking  a  spacecraft  with  another 
moving  craft. 

Y.  COGNITION  OF  BEHAVIORAL  UNITS: 

1.  Recognizing  a  simple,  distinct,  behavioral 
action.  Perceiving  a  punch  in  the  face  or  a 
salute. 

2.  Recognizing  a  potentially  ambiguous  behavioral 
action.  Identifying  a  smile,  a  wink,  or  a  nod. 

3.  Recognizing  a  complex  behavioral  action  in 
varying  degrees  of  ambiguity.  Identifying  a 
leer,  a  frown,  or  a  ’’menacing  look.” 

Z.  COGNITION  OF  BEHAVIORAL  CLASSES: 

1.  Classifying  behavioral  items  by  a  few  simple  or 
readily  apparent  attributes.  Given  a  defined 
set  of  simple  actions,  which  are  gestures  or 
which  are  postures. 

2.  Classifying  a  potentially  ambiguous  behavioral 
action.  Given  a  set  of  actions,  which  indicate 
hostility  or  which  indicate  happiness. 

3.  Classifying  complex  behavioral  activities  on 
several  potentially  ambiguous  or  not  readily 
apparent  dimensions.  Classifying  a  dance  as 
joyful  or  atmosphere  at  a  meeting  as  tense. 

AA.  COGNITION  OF  BEHAVIORAL  RELATIONS: 

1.  Comprehending  simple  relationships  among  a  few 
distinct  behavioral  actions  or  situations. 

Waving  when  one  says  good-bye  or  that  smiling 
people  are  generally  happy. 

2.  Comprehending  relationships  between  potentially 
ambiguous  behavioral  actions  or  situations. 
Understanding  scared  people  tend  to  have  higher 
heart-rates  and  blood  pressures 
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or  that  dessert  usually  follows  dinner  (in  many 
Western  cultures) . 

3.  Comprehending  complex  relationships  among 
potentially  ambiguous  behavioral  actions  or 
situations.  Understanding  how  to  act  towards 
"the  boss"  at  an  office  party. 

BB.  COGNITION  OF  BEHAVIORAL  SYSTEMS: 

1.  Perceiving  the  interactions  and  acceptable 
behavior  in  a  simple  social  system. 

Understanding  basic  standards  of  sportsmanship 
and  fair  play  during  a  game  of  checkers. 

2.  Perceiving  the  interactions  and  acceptable 
behavior  in  a  moderately  complex  social  system. 
Proper  etiquette  at  a  formal  dinner  or 
acceptable  conduct  on  the  basketball  court  by  a 
player  for  a  given  situation. 

3.  Perceiving  the  interactions  and  acceptable 
behavior  in  a  complex  social  system  with  varying 
degrees  of  ambiguity.  Understanding  what  is 
actually  being  communicated  ("reading  between 
the  lines")  during  a  negotiation. 

CC.  COGNITION  OF  BEHAVIORAL  TRANSFORMATIONS: 

1.  Reinterpreting  a  simple  action  or  situation, 
based  on  new  information,  to  alter  its 
behavioral  value.  Determining  how  a  particular 
action  (i.e.,  saying  goodbye)  might  differ 
between  a  mother  and  son  or  a  father  and  son. 

2.  Reinterpreting  a  possibly  ambiguous  action  or 
situation,  based  on  new  information,  to  alter 
its  behavioral  value.  Based  on  the  size  ($1, 

$20,  etc.)  of  the  bill  found,  a  person  will 
decide  whether  or  not  to  turn-in  cash  found  in  a 
business  place. 

3.  Reinterpreting  a  complex  activity  or  situation, 
based  on  new  information,  to  alter  its 
behavioral  value.  Despite  close  military  and 
economic  ties,  two  allies  sever  relations 
because  of  new  ideological  differences. 

DD.  COGNITION  OF  BEHAVIORAL  IMPLICATIONS: 

1.  Recognizing  the  obvious  consequences  suggested 
by  a  simple  behavioral  action  or  situation.  If 
the  phone  rings  you  answer  it  or  when  you  wish 
to  enter  a  room  you  knock  if  the  door  is  closed. 

2.  Recognizing  the  possible  consequences  suggested 
in  a  behavioral  action.  A  smile  may  signify 
happiness  or  kicking  dirt  on  an  umpire's  shoes 
usually  results  in  ejection  from  the  game. 
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3.  Understanding  the  possible  consequences  in  an 
ambiguous  behavioral  activity  or  situation. 
Determining  if  a  seller  will  accept  a  particular 
bid  or  if  the  body  language  of  a  person  in  a 
bank  carrying  a  large  bag  suggests  a  criminal 
intent . 
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SECTION  2.  MEMORY.  The  commitment  of  cognized  information 
to  storage  for  later  use.  The  information  is  stored  in  the 
same  form  as  cognized  and  is  accessible  through  the  same  cues 
connected  to  it  when  stored.  Learning  is  encompassed  in  this 
operation.  Memorizing  a  telephone  number,  an  acquaintance's 
face,  or  the  words  to  the  "Star  Spangled  Banner,"  are  all 
examples  of  the  memory  operation. 

YES _  (answer  remaining  questions)  NO _  (proceed  to 

next  section) 

F  -  C 

A.  MEMORY  OF  VISUAL  UNITS: 

_  -  _  1.  Storing  a  simple,  distinct,  visual  form.  Saving 

a  square  or  a  color. 

_  -  _  2.  Storing  a  moderately  complex  visual  form  along 

gross  dimensions.  Saving  an  object  called 
"bank"  or  and  object  called  "jeep." 

_  -  _  3.  Storing  a  complex  visual  form  in  varying  degrees 

of  background  clutter.  Saving  the  image  of  a 
enemy  fighter's  radar  signature  or  the 
nuclear-magnetic  resonance  image  of  a  fractured 
femur. 

F  -  C 

B.  MEMORY  OF  VISUAL  CLASSES: 

_  -  _  1.  Storing  visual  items  categorized  by  a  few  simple 

or  readily  apparent  attributes.  Saving  objects 
in  classes  such  as  blue  items  or  wheeled  items, 
etc. 

_  -  _  2.  Storing  visual  forms  categorized  on  several 

moderately  ambiguous  or  not  readily  apparent 
dimensions.  Saving  objects  as  leaves,  vintage 
automobiles,  etc. 

_  -  _  3.  Storing  complex  visual  forms  categorized  on 

ambiguous  or  esoteric  factors.  Saving  images  of 
paintings  as  examples  of  cubism,  efficient  house 
plans,  or  ugly  colors. 

F  -  C 

C.  MEMORY  OF  VISUAL  RELATIONS: 

_  -  _  1.  Saving  simple  relationships  among  a  few  distinct 

visual  forms.  Recording  that  a  stop  sign, 
apple,  and  cherry  are  examples  of  the  color  red. 

_  -  _  2.  Saving  more  involved  relationships  among  a  few 

visual  forms.  Recording  that  a  one  object  is 
"closer"  than  another  or  an  object  is  "to  the 
right"  of  another  object. 

_  -  _  3.  Saving  complex  relationships  among  potentially 

ambiguous  visual  forms  and  dimensions.  Recording 
that  a  Porsche  and  a  Corvette  are  related  since 
they  look  like  "sports  cars"  or  that  some 
symptoms  (such  as  a  skin  rash)  may  look  the  same 
for  different  illnesses. 
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D.  MEMORY  OF  VISUAL  SYSTEMS: 

1.  Recording  the  static  arrangement  and  position  a 
few  simple  visual  forms.  Storing  positions  of 
appliances  in  a  kitchen  or  the  position  of  an 
object  in  a  two-dimensional  plane. 

2.  Recording  the  arrangement  and  spatial  position 
of  several  moderately  complex  objects. 

Recalling  play  tendency  based  upon  a  given 
football  formation  or  the  position  of  all 
aircraft  in  a  holding  pattern. 

3.  Recording  the  arrangement  and  spatial  positions 
of  members  of  a  complex  system  of  visual  form  in 
varying  degrees  of  background  clutter. 

Recalling  the  arrangement  and  positions  of 
organs  in  the  abdomen. 

E.  MEMORY  OF  VISUAL  TRANSFORMATIONS: 

1.  Saving  result  simple,  significant  changes  on  a 
simple  visual  object.  Storing  the  image  of  a 
block  after  rotating  it  180  degrees  or 
designating  an  object  as  now  red  in  color 
instead  of  blue. 

2.  Understanding  the  result  after  several  changes 
are  imposed  on  a  moderately  complex  visual 
scene.  Selecting  the  correct  piece  to  complete 
a  puzzle  or  the  arrangement  of  appliances  in  a 
kitchen  has  been  changed. 

3.  Understanding  the  impact  of  subtle  changes  in  a 
complex  visual  form.  Storing  a  man's  face  after 
aging  30  years  or  recording  how  design  changes 
will  affect  the  visual  appearance  of  a  vehicle. 

F.  MEMORY  OF  VISUAL  IMPLICATIONS: 

1.  Storing  the  obvious  consequences  inherent  in  an 
unambiguous  visual  scene.  Recalling  a  simple 
circuit  diagram  or  directions  to  a  particular 
place. 

2 .  Storing  the  learned  consequences  inherent  in  a 
possibly  ambiguous  visual  scene.  Recalling 
alternate  routes  to  get  from  the  airport  to  your 
house  or  recalling  the  standards  for  determining 
if  a  piece  of  fruit  is  Grade  "A,'’  MB,M  or 
discarded. 

3.  Storing  the  possible  learned  consequences  in 
an  ambiguous  visual  scene.  Recalling  which 
air-to-air  maneuvers  are  most  effective  for  a 
given  aerial  combat  situation  or  what  ailments 
are  suggested  by  red,  mottled  skin. 
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G.  MEMORY  OF  AUDITORY  UNITS: 

1.  Storing  a  simple,  distinct,  auditory  stimulus. 
Recalling  the  sound  of  a  school  bell  or  a  car 
horn. 

2.  Storing  a  moderately  complex  auditory  stimulus 
along  gross  dimensions.  Recalling  a  specific 
tone  from  several  candidate  tones  or  recalling 
examples  of  rock  music. 

3.  Storing  a  complex  auditory  stimulus  in  varying 
degrees  of  background  clutter.  Recalling  the 
way  individual  words  sound. 

H.  MEMORY  OF  AUDITORY  CLASSES: 

I.  Storing  classifications  of  auditory  stimuli  on  a 
few  simple  or  readily  apparent  attributes. 
Storing  all  tones  less  than  5,000  Hz  or  sounds 
between  10  and  10,000  decibels. 

2.  Storing  classifications  of  auditory  stimuli  on 
several  moderately  ambiguous  or  not  readily 
apparent  dimensions.  Storing  a  dog's  bark  as 
loud  or  soft  or  a  car  horn  as  either  high  or 
low-pitched. 

3.  Storing  classifications  of  complex  auditory 
stimuli  on  ambiguous  or  esoteric  factors. 

Storing  examples  of  music  composed  by  Chopin  or 
submarine  classifications  based  on  the  acoustic 
signature  of  its  screws  under  water. 

I.  MEMORY  OF  AUDITORY  RELATIONS: 

1.  Saving  simple  relationships  among  a  few  distinct 
auditory  stimuli.  Storing  the  relative 
frequency  difference  of  two  tones. 

2 .  Saving  more  involved  relationships  among  a  few 
auditory  stimuli.  Storing  the  relationships 
between  frequency  spectrums  and  harmonics  of 
several  sounds. 

3.  Saving  complex  relationships  among  potentially 
ambiguous  auditory  stimuli  and  dimensions. 
Storing  a  variety  of  relationships  among  several 
sounds  such  as  sound  quality,  tonal 
characteristics,  or  "pleasantness." 

J.  MEMORY  OF  AUDITORY  SYSTEMS: 

l.  Recording  the  arrangement  and  position  of  a  few 
simple  sounds  that  form  a  unique  whole. 
Remembering  the  position  of  a  ticking  clock  on 
the  nightstand  beside  your  bed  or  knowing  the 
position  of  a  passing  motorist  from  hearing  her 
vehicle. 


2.  Recording  the  arrangement  and  position  of 
several  moderately  complex  sounds.  Remembering 
where  certain  pieces  of  heavy  equipment  are 
located  based  on  the  relative  loudness  of  each 
machine . 

3.  Recording  the  arrangement  and  position  of 
various  stimuli  in  a  complex  auditory  system 
with  varying  degrees  of  background  clutter. 
Remembering  the  position  of  voices  of  various 
people  seated  around  you  at  an  event. 

K.  MEMORY  OF  AUDITORY  TRANSFORMATIONS: 

1.  Saving  the  impact  of  changes  on  a  simple 
auditory  stimulus  after  some  specific  changes 
occur.  Recalling  how  a  squeaky  door  sounds 
after  oiling  it. 

2.  Saving  the  result  of  several  changes  imposed  on 
a  moderately  complex  auditory  arrangement. 
Storing  the  impact  of  changing  the  tempo  and  key 
signature  on  a  piece  of  music. 

3.  Saving  the  impact  of  subtle  changes  in  a  complex 
auditory  system.  Storing  verbal  commands  made 
by  a  pilot  under  high-g  stress  or  interpreting 
words  that  are  sung  rather  than  spoken. 

L.  MEMORY  OF  AUDITORY  IMPLICATIONS: 

1.  Storing  the  obvious  consequences  inherent  in  an 
unambiguous  auditory  stimulus.  Recalling  to 
fasten  a  safety  belt  when  the  car  "seat-belt 
buzzer"  sounds  or  remembering  that  a  ringing 
alarm  clock  signals  time  to  get-up. 

2.  Storing  the  learned  consequences  in  a  possibly 
ambiguous  auditory  system.  Remembering  how  an 
asthma  patient  sounds  just  before  having  an 
asthma  attack  or  how  a  properly  tuned  engine 
sounds . 

3.  Storing  the  learned  consequences  in  an  ambiguous 
auditory  system.  Recalling  what  to  do  to 
relieve  a  particular  sound  of  a  baby's  crying 
(i.e.,  a  "hungry"  cry  or  "change  me"  cry). 

M.  MEMORY  OF  SYMBOLIC  UNITS: 

1.  Storing  a  simple,  distinct,  symbol,  storing  a 
plus  (+)  sign  or  an  ASCII  character. 

2.  Storing  a  moderately  complex  symbol.  Recalling 
an  oil  symbol  in  an  auto  or  remembering  the 
company  associated  with  a  particular  logo. 

3.  Storing  a  complex  symbol  for  recognition  in 
varying  degrees  of  background  clutter.  Recalling 
handwritten  words  or  a  specific  Chinese 
pictograph. 
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N.  MEMORY  OF  SYMBOLIC  CLASSES: 

1.  Storing  symbolic  classifications  by  a  few  simple 
or  readily  apparent  attributes.  Saving 
groupings  of  correctly  spelled  words  or  Arabic 
numerals . 

2.  Storing  symbolic  classifications  on  several 
moderately  ambiguous  or  not  readily  apparent 
dimensions.  Storing  military  rank  insignia  or 
national  flags. 

3.  Storing  complex  symbols  classified  on  ambiguous 
or  esoteric  factors.  Recalling  patriotic 
symbols  or  symbols  of  cultural  significance. 

O.  MEMORY  OF  SYMBOLIC  RELATIONS: 

1.  Saving  simple  relationships  among  a  few  distinct 
symbols.  Recalling  that  "26"  is  greater  than 
"IS’'  or  "+M  operations  are  performed  before  "xH 
operations. 

2.  Saving  more  involved  relationships  among  a  few 
symbols.  Remembering  a  silver  star  (symbolizing 
the  rank  of  general)  outranks  a  two  silver  bars 
(symbolizing  the  rank  of  captain) . 

3.  Saving  complex  relationships  among  several 
symbols.  Recalling  which  elements  of  the 
periodic  table  can  be  combined  (within  the  laws 
of  physics) . 

P.  MEMORY  OF  SYMBOLIC  SYSTEMS: 

1.  Recording  three  or  more  simple  symbols  composing 
a  collective  whole.  Saving  telephone  or  Social 
Security  numbers  or  letters  forming  words. 

2.  Recording  the  integration  of  many  moderately 
complex  symbols.  Storing  words  organized  in 
simple  sentences  or  symbols  used  in  a  military 
information  distribution  system. 

3.  Recording  the  integration  of  many  complex 
symbols.  Saving  a  highly-complex  military  code 
or  storing  genetic  code  information. 

Q.  MEMORY  OF  SYMBOLIC  TRANSFORMATIONS: 

1.  Recording  the  impact  of  changes  on  simple 
symbols  after  some  specific  changes  occur. 
Updating  symbols  on  a  map  display  as  information 
of  their  disposition  is  obtained 

or  recalling  translations  of  individual  words 
from  the  Japanese  to  English  language. 

2 .  Recording  the  impact  of  several  changes  imposed 
on  a  moderately  complex  symbol  set.  Recalling 
typical  letter  sequences  to  arrange  scrambled 
letters  to  form  a  meaningful  word  or  the 
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remembering  the  notes  for  a  given  position  in 
the  bass  clef  versus  the  treble  clef. 

3.  Recording  the  impact  of  subtle  changes  in  a 
complex  symbol  set.  Recalling  the  audible 
counterparts  of  written  words  for  reading  aloud 
or  translating  from  Russian  to  English. 

R.  MEMORY  OF  SYMBOLIC  IMPLICATIONS: 

1.  Storing  the  obvious  information  inherent  in  an 
unambiguous  symbol.  Recalling  a  red  light  on  a 
traffic  light  implies  "stop"  or  implies 
subtraction  of  two  quantities. 

2.  Storing  the  information  contained  in  a  possibly 
ambiguous  symbol.  Recording  the  implication 
that  inclusion  of  the  first  three  numbers  of  a 
telephone  number  (i.e.,  (Ill)  555-2345)  means 
the  number  is  a  long  distance  call. 

3.  Storing  information  contained  in  an  ambiguous 
symbol.  Recording  the  salient  aspects  of  an 
unknown  symbol  for  later  evaluation  (i.e.,  "I'm 
not  sure  what  this  means,  but  I'll  remember  it 
in  case  I  see  it  in  the  future) . 

S.  MEMORY  OF  SEMANTIC  UNITS: 

1.  Storing  the  meaning  of  a  simple  idea  or  concept. 
Saving  the  sound  of  a  particular  note  or  the 
meaning  of  basic  domain  elements  such  as  "dog," 
"car,"  "bed,"  etc. 

2.  Storing  the  meaning  of  a  moderately  complex  idea 
or  concept.  Recalling  the  meaning  of 
less-general  terms  in  the  domain  such  as  "too 
hot,"  "sell-short,"  or  other  potentially 
ambiguous  concepts  or  jargon. 

3.  Storing  the  meaning  or  interpretation  of  a 
complex  idea  or  concept.  Remembering  meanings 
of  abstract  terms  like  "love,"  "beautiful,"  or 
other  highly  subjective  concepts. 

T.  MEMORY  OF  SEMANTIC  CLASSES: 

1.  Encoding  similar  concepts  classified  on  a  few 
simple  or  readily  apparent  attributes.  Saving 
groups  of  words  that  have  like  meanings  or 
objects  that  share  an  obvious,  common  purpose. 

2.  Encoding  concepts  classified  on  several 
moderately  ambiguous  or  not  readily  apparent 
dimensions.  Storing  similar  types  of 
architectural  design  concepts  or  a  grouping 
football  plays  for  particular  down  and  distance 
situations  (i.e.,  First  and  10  plays  are 
different  than  Third  and  Long  plays) . 
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3.  Encoding  complex  concepts  classified  on 

ambiguous  or  esoteric  factors.  Saving  various 
types  of  speeches  for  the  President  based  on  the 
type  of  public  opinion  they  are  intended  to 
evoke  (i.e.,  some  speeches  used  for  patriotic 
occasions,  others  for  national  crisis,  etc.). 

U.  MEMORY  OF  SEMANTIC  RELATIONS: 

1.  Saving  simple,  concrete  relationships  among  a 
few  concepts  or  situations.  Storing  the  link 
between  "start"  and  "finish"  or  other  simple 
antonyms  or  synonyms. 

2.  Saving  less  certain  relationships  among  a  few 
concepts.  Recalling  the  general  positive 
correlation  of  personal  income  and  standard  of 
living  or  the  general  relationship  of  the  of 
"husband"  to  "wife." 

3.  Saving  complex  relationships  among  potentially 
ambiguous  concepts  or  ideas.  Storing  knowledge 
of  the  concept  of  "communism"  relative  to  the 
concept  of  "socialism." 

V.  MEMORY  OF  SEMANTIC  SYSTEMS: 

1 .  Recording  the  integration  of  three  or  more 
simple,  unambiguous  concepts  which  comprise  a 
unique  whole.  Storing  the  rules  of  chess  or 
basic  accounting. 

2.  Recording  the  integration  of  several  moderately 
complex  concepts  in  potentially  ambiguous 
contexts.  Recalling  key  meanings  in  an 
interoffice  memo. 

3 .  Recording  the  integration  of  many  complex 
concepts  in  generally  ambiguous  contexts. 
Remembering  the  central  themes  of  "Das  Kapital" 
or  the  meaning  of  a  teacher's  lecture. 

W.  MEMORY  OF  SEMANTIC  TRANSFORMATIONS: 

1.  Recalling  the  impact  of  simple  changes  imposed 
on  basic  concepts.  Saving  the  redefinition  of  a 
new  use  for  a  simple  item  (i.e.,  a  child 
remembering  a  phone  book  can  be  useful  as  a 
step-stool) . 

2.  Recalling  the  impact  of  several  changes  imposed 
on  a  moderately  complex  conceptual  system. 

Saving  the  revision  that  an  aircraft  is 
experiencing  mechanical  problems  and  will  be 
unable  to  depart  as  scheduled. 

3.  Understanding  the  impact  of  subtle  or  major 
changes  in  the  interpretation  of  a  complex 
conceptual  system  or  situation.  Saving  the 
changes  needed  to  make  an  old  social  welfare 


102 


program  responsive  to  the  new  needs  of  welfare 
recipients . 

X.  MEMORY  OF  SEMANTIC  IMPLICATIONS: 

1.  Storing  the  obvious  consequences  inherent  in  a 
unambiguous  concept  or  situation.  Recalling  the 
consequences  associated  with  the  word  "hot"  or 
"danger. " 

2.  Storing  the  learned  consequences  inherent  in  a 
possibly  ambiguous  conceptual  system.  Recalling 
the  tools  used  in  performing  a  given  job  or  the 
procedures  required  to  alert  the  maintenance 
crew  an  aircraft  requires  their  attention. 

3.  Storing  the  learned  possible  consequences 
suggested  in  an  ambiguous  conceptual  system  or 
situation.  Recalling  the  necessary  steps  taken 
(and  other  steps  considered)  in  past  (but 
slightly  different)  situations  to  alleviate  a 
patient's  suffering. 

Y.  MEMORY  OF  BEHAVIORAL  UNITS: 

1.  Storing  a  simple,  distinct,  behavioral  action. 
Recording  a  smile,  a  wink,  a  salute,  or  crying. 

2.  Storing  a  moderately  complex  behavioral  action 
with  possible  ambiguity.  Recalling  a  "come 
here"  gesture  or  storing  the  different  ways  a 
mother  can  say  "no." 

3.  Storing  a  complex  behavioral  action  with  some 
degree  of  ambiguity.  Recalling  how  an  "icy 
gaze"  or  a  "sheepish  grin"  appears. 

Z.  MEMORY  OF  BEHAVIORAL  CLASSES: 

1.  Encoding  behavioral  items  classified  on  a  few 
simple  or  readily  apparent  attributes.  Storing 
actions  which  indicate  a  number  or  specify  a 
direction. 

2.  Encoding  moderately  complex  behavioral 
activities  classified  on  potentially  ambiguous 
dimensions.  Storing  actions  which  suggest 
happiness  or  illness. 

3.  Encoding  complex  behavioral  activities 
classified  on  several  potentially  ambiguous  or 
not  readily  apparent  dimensions.  Storing 
ballets  grouped  according  to  their  moods  (i.e., 
somber  or  joyous)  or  recalling  feelings 
associated  with  people  based  upon  their 
personality. 

AA.  MEMORY  OF  BEHAVIORAL  RELATIONS: 

1.  Saving  simple  relationships  among  a  few  distinct 
behavioral  actions  or  situations.  Remembering 
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when  it  is  appropriate  to  salute  or  to  stand 
when  the  national  anthem  is  played. 

2.  Saving  moderately  complex  relationships  among 
potentially  ambiguous  behavioral  actions. 

3.  Saving  complex  relationships  among  ambiguous 
behavioral  actions  or  situations.  Remembering 
political  or  ideological  beliefs  of  certain 
individuals  and  how  to  relate  to  them  to 
minimize  conflict. 

BB.  MEMORY  OF  BEHAVIORAL  SYSTEMS: 

1.  Recording  the  acceptable  interactions  and 
activities  in  a  simple  social  system.  Recalling 
the  proper  way  to  behave  when  playing  with  other 
children  (no  hitting,  no  biting,  sharing  toys, 
etc . )  . 

2.  Recording  the  interactions  and  acceptable 
actions  in  a  moderately  complex  social  system. 
Recalling  the  proper  fork  to  use  at  a  formal 
dinner. 

3.  Recording  the  interactions  and  acceptable 
actions  in  a  complex  social  system  with  varying 
degrees  of  ambiguity.  Saving  environmental  cues 
(i.e.,  the  behaviors  of  others)  that  suggest 
acceptable  norms  and  behaviors  in  a  foreign 
setting. 

CC.  MEMORY  OF  BEHAVIORAL  TRANSFORMATIONS:  - -> 

1.  Recording  the  reinterpretation  of  a  simple 
action  that  alters  its  behavioral  value.  After  a 
child  is  spanked  for  writing  on  the  livingroom 
wall,  she  now  interprets  the  act  of  writing  on 
the  wall  as  unacceptable  behavior. 

2.  Recording  the  reinterpretation  of  a  moderately 
complex  behavioral  action  or  situation.  An 
automobile  driver  who  habitually  exceeds  the 
speed  limit  sees  graphic  portrayals  of  accidents 
due  to  excessive  speed  and  then  updates  her 
belief  system  to  view  speeding  as  inappropriate. 

3.  Recording  the  reinterpretation  of  a  complex 
behavioral  activity  that  alters  its  behavioral 
value.  Updating  the  previously  negative 
cultural  stereotypes  held  by  an  individual  about 
certain  ethnic  groups  after  positively 
interacting  with  members  of  these  groups. 

DD.  MEMORY  OF  BEHAVIORAL  IMPLICATIONS: 

1.  Storing  the  learned  consequences  explicit  in  a 
non-complex  behavioral  situation  or  action. 
Recalling  that  specific  actions  (such  as  crying) 
result  in  specific  outcomes  (comforting  from  a 
parent) . 
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2.  Storing  the  learned  consequences  suggested  in  a 
generally  unambiguous  behavioral  action  or 
situation.  Recalling  a  smile  may  signify 
happiness  or  amusement  or  striking  someone  often 
results  in  retaliation. 

3.  Storing  the  learned  consequences  in  an  ambiguous 
behavioral  activity  or  situation.  Remembering 
which  sales  pitches  result  in  the  greatest 
success  based  upon  the  probable  background  of 
the  buyer  or  how  to  calm  an  unruly  crowd  at  a 
rock  concert. 
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SECTION  3.  DIVERGENT  PRODUCTION.  This  operation  involves 
the  generation  of  alternative  information  elements  from  the 
memory  store,  in  either  exact  or  modified  form,  where  the 
emphasis  is  placed  upon  variety,  quantity,  and  relevance  of 
the  alternatives  to  satisfy  the  conditions  of  a  particular 
situation.  Also  called  inductive  reasoning.  Naming  objects 
that  are  both  heavy  and  mobile  and  brainstorming  alternative 
solutions  to  a  problem  are  examples  of  divergent  production. 
Engaging  in  creative  activity  of  any  kind  relies  heavily  on 
the  mental  operation  of  divergent  production. 

YES _  (answer  remaining  questions)  NO _  (proceed  to 

next  section) 

F  -  C 

A.  DIVERGENT  PRODUCTION  OF  VISUAL  UNITS: 

_  1.  Generating  simple,  distinct,  visual  forms  to 

satisfy  a  simple  requirement.  Producing  a 
triangle,  square  and  circle  for  the  requirement 
of  "basic  shapes." 

_  2.  Generating  a  moderately  complex  visual  form 

given  some  possibly  ambiguous  dimension (s) . 
Producing  several  undetailed  images  of  shops, 
passers-by,  and  autos  for  the  requirement  of 
"things  seen  on  a  street." 

_  3.  Generating  a  complex  visual  form  to  satisfy 

potentially  ambiguous  requirement (s) .  Producing 
a  number  of  highly-  detailed  images  to  satisfy 
the  requirement  of  "landscapes  in  the 
impressionistic  style  of  Monet." 

C 

B.  DIVERGENT  PRODUCTION  OF  VISUAL  CLASSES: 

_  1.  Generating  alternative  ways  of  classifying 

simple  visual  items  by  readily  apparent 
attributes.  Classifying  a  group  of  shapes  by 
size,  number  of  sides,  or  color. 

_  2.  Generating  alternative  ways  of  classifying 

moderately  complex  visual  forms  on  moderately 
ambiguous  or  not  readily  apparent  dimensions. 
Classifying  a  group  of  buildings  by  age  or 
"usable"  square-footage. 

_  3.  Generating  alternative  ways  of  classifying 

complex  visual  forms  by  ambiguous  or  esoteric 
attributes.  Classifying  cities  by  "quality  of 
life,"  "attractiveness,"  or  "desirability." 

F  -  C 

C.  DIVERGENT  PRODUCTION  OF  VISUAL  RELATIONS: 

_  -  _  1.  Producing  a  variety  of  simple  relationships 

among  a  few  distinct  visual  forms.  Listing  a 
number  of  relationships  between  a  pair  of 
apples,  such  as  apple  1  is  redder  and  bigger 
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than  apple  2,  while  apple  2  is  rounder  and 
glossier  than  apple  1. 

2.  Producing  a  variety  of  moderately  complex 
relationships  among  a  few  visual  forms. 
Specifying  a  certain  auto  looks  "boxy"  compared 
to  another. 

3.  Producing  complex  relationships  among 
potentially  ambiguous  visual  forms  and 
dimensions.  Describing  one  picture  as  being 
"warmer  and  more  appealing"  than  another  or 
examining  a  patient  and  relating  his  physical 
appearance  to  understandable  terms  for 
discussing  the  condition  with  a  colleague  (i.e., 
"He  looked  like  ..."). 

D.  DIVERGENT  PRODUCTION  OF  VISUAL  SYSTEMS: 

1.  Inventing  a  number  of  simple  composites  using  a 
few  simple  visual  forms  to  satisfy  a  definite 
requirement.  Given  three  lines,  one  arc,  and  a 
square,  produce  as  many  pictures  of  real  objects 
as  possible. 

2.  Inventing  a  number  of  moderately  complex  objects 
using  more  complex  visual  forms  to  satisfy  some 
requirement.  Using  several  fruit  and  flower 
images,  design  different  wallpaper  patterns. 

3.  Inventing  a  number  of  complex  objects  using 
complex  visual  forms  to  satisfy  an  ambiguous 
requirement.  Developing  alternative  versions  of 
a  painting  or  sculpture. 

E.  DIVERGENT  PRODUCTION  OF  VISUAL  TRANSFORMATIONS: 

1.  Creating  alternative  ways  of  processing  simple 
visual  information  to  satisfy  a  distinct 
requirement.  Describing  the  possible  ways  of 
changing  a  simple  scene  to  change  its  meaning. 

2.  Creating  alternative  ways  of  processing 
moderately  complex  visual  information  to  satisfy 
some  requirements.  Specifying  ways  of  changing  a 
dress  pattern  to  make  the  dress  "more  formal." 

3.  Creating  alternative  ways  of  processing  complex 
visual  information  to  satisfy  ambiguous 
requirements.  Changing  the  packaging  of  a 
product  to  make  it  more  "appealing"  to  buyers  or 
a  pilot  inventing  alternative  routes  to 
efficiently  complete  a  skywritten  message. 

F.  DIVERGENT  PRODUCTION  OF  VISUAL  IMPLICATIONS: 

1.  Creating  various  alternatives  suggested  by 

distinct  visual  information  to  conform  to  exact 
requirements.  Producing  alternate  colors  to  use 
for  representing  a  specific  state  or  condition. 
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2.  Creating  various  alternatives  suggested  by 
possibly  ambiguous  visual  information  to  satisfy 
general  requirements.  Producing  alternate  cake 
decorations  for  a  boy's  birthday  or  for  a  girl's 
birthday. 

3.  Creating  various  alternatives  suggested  by 
ambiguous  visual  information  to  satisfy 
ambiguous  requirements.  Developing  several 
interior  design  schemes  to  blend  with  the 
surrounding  architecture  or  designing  a  space 
laboratory  living  quarters  to  be  both 
functionally  efficient  and  esthetically 
pleasing. 

G.  DIVERGENT  PRODUCTION  OF  AUDITORY  UNITS: 

1.  Generating  a  variety  of  simple,  distinct, 
auditory  stimuli  for  a  specific  requirement. 
Specify  sounds  heard  in  a  school  gym  or  loud 
noises  at  an  airport. 

2 .  Generating  a  number  of  moderately  complex 
auditory  stimuli  to  satisfy  some  general 
requirements.  Produce  examples  of  sounds  from  a 
violin  or  examples  of  animal  sounds. 

3.  Generating  a  variety  of  complex  auditory  stimuli 
to  satisfy  ambiguous  requirements.  Producing  the 
word  "exit"  as  spoken  in  different  languages. 

H.  DIVERGENT  PRODUCTION  OF  AUDITORY  CLASSES : 

I.  Generating  alternative  ways  of  classifying 
auditory  stimuli  by  a  few  simple  or  readily 
apparent  attributes.  Grouping  sounds  by  exact 
frequency  ranges  or  loudness  determined  in 
decibels. 

2.  Generating  alternative  ways  of  classifying 
auditory  stimuli  on  several  moderately  ambiguous 
or  not  readily  apparent  dimensions.  Given  a  set 
of  dog  barks,  specify  possible  groups  such  as 
loud  or  soft,  high  or  low-pitched,  or  "mean 
dogs"  or  "nice  dogs." 

3.  Generating  alternative  ways  of  classifying 
complex  auditory  stimuli  on  ambiguous  or 
esoteric  factors.  Hearing  a  conversation  and 
describing  dimensions  for  classifying  it  as 
friendly,  hostile,  or  non-commital . 

I.  DIVERGENT  PRODUCTION  OF  AUDITORY  RELATIONS: 

1.  Producing  a  number  of  simple  relationships  among 
a  few  distinct  auditory  stimuli.  Generating 
basic  relationships,  such  as  loudness,  key,  or 
pitch  among  a  set  of  tones. 
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2 .  Producing  more  involved  relationships  among  a 
few  auditory  stimuli.  Generating  alternative 
relationships  among  various  "meows"  of  a  pet  cat 
relative  to  the  cat's  desire  to  eat,  go  out, 
etc. 

3.  Producing  complex  relationships  among 
potentially  ambiguous  auditory  stimuli  and 
dimensions.  Generating  relationships  between 
expected  sounds  of  a  cardiac 

patient's  heart  and  the  observed  sounds. 

J.  DIVERGENT  PRODUCTION  OF  AUDITORY  SYSTEMS: 

1.  Generating  a  few  simple  sounds  form  several 
unique  systems  that  meet  specific  requirements. 
Combining  several  tones  to  produce  different 
warning  signals. 

2 .  Generating  several  moderately  complex  auditory 
systems  to  satisfy  general  requirements. 
Specifying  various  short  musical  melodies  for 
use  as  doorbell  tunes. 

3 .  Generating  complex  auditory  systems  to  satisfy  a 
number  of  ambiguous  requirements.  Composing  new 
theme  music  for  a  motion  picture. 

K.  DIVERGENT  PRODUCTION  OF  AUDITORY 

TRANSFORMATIONS : 

1.  Envisioning  the  impact  of  specific  changes  upon 
a  simple  auditory  stimulus.  Imagining  how  an  a 
particular  alarm  might  be  changed  to  enhance  its 
audibility  or  attention-  getting  qualities. 

2.  Envisioning  the  impact  of  several  changes 
imposed  on  a  moderately  complex  auditory  system. 
Imagining  the  impact  of  changing  the 

key  and  tempo  of  an  musical  selection  to  adapt 
it  to  a  new  audience  (i.e.,  adapting  pop  tunes 
for  "easy  listening"  formats) . 

3 .  Envisioning  the  impact  of  subtle  changes  in  a 
complex  auditory  system.  Imagining  how  an 
automobile  engine  sound  will  change  based  upon 
changes  in  the  idle  speed,  carburetor 
adjustment,  or  other  fine-tuning  adjustments. 

L.  DIVERGENT  PRODUCTION  OF  AUDITORY  IMPLICATIONS: 

1.  Generating  several  alternative  consequences 
perceived  to  be  inherent  in  a  distinct  auditory 
stimulus.  Inferring  possible  meanings  of  being 
"honked  at"  in  traffic  by  a  passing  motorist 
(i.e.,  lights  are  on,  taillights  are 
inoperative,  car  is  on  fire,  etc.). 

2.  Generating  various  alternative  consequences 
perceived  to  be  inherent  in  a  possibly  ambiguous 
auditory  stimulus.  Producing  several  hypotheses 
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about  the  origin  of  a  "strange  sucking  sound" 
heard  by  a  doctor  during  a  routine  examination 
of  a  patient's  lungs. 

3.  Generating  various  alternative  consequences 
perceived  to  be  inherent  in  an  ambiguous 
auditory  system.  Imagining  the  possible  origins 
of  "cosmic  noise"  or  if  a  particular  tone  of 
voice  is  "threatening." 

M.  DIVERGENT  PRODUCTION  OF  SYMBOLIC  UNITS: 

1.  Generating  a  number  of  simple,  distinct,  symbols 
to  meet  a  specific  requirement.  Producing  as 
many  three  letter  words  ending  in  "id"  as 
possible. 

2.  Generating  a  number  of  moderately  complex 
symbols  to  meet  general  requirements.  Producing 
several  logos  to  represent  a  new  product. 

3 .  Generating  a  number  of  complex  symbols  to  meet 
ambiguous  or  conflicting  requirements.  Imagining 
alternative  icons  for  representing  human 
activities  (such  as  the  standard  international 
traffic  symbols) . 

N.  DIVERGENT  PRODUCTION  OF  SYMBOLIC  CLASSES: 

1.  Producing  alternative  classifications  of  symbols 
on  obvious  dimensions.  Suggesting  ways  of 
grouping  simple  symbols  such  as  numbers  with 
curves  (i.e.,  2,3,5)  or  no  curves  (i.e.,  1,4). 

2.  Producing  alternative  classifications  of  symbols 
on  several  moderately  ambiguous  or  not  readily 
apparent  dimensions.  Suggesting  new 

ways  of  grouping  punctuation  marks  (i.e.,  select 
those  typically  used  by  accountants,  or  those 
with  several  meanings,  etc.). 

3.  Producing  alternative  classifications  of  complex 
symbols  on  ambiguous  or  esoteric  factors. 
Suggesting  new  ways  of  grouping  patriotic 
symbols  (i.e.,  symbols  that  became  popular  in 
WWII  but  continue  to  endure,  etc.)  or  symbols  of 
cultural  significance  (i.e.,  symbols  that  people 
can  walk  in,  etc.). 

O.  DIVERGENT  PRODUCTION  OF  SYMBOLIC  RELATIONS: 

1.  Producing  a  number  of  simple  relationships  among 
a  few  distinct  symbols  to  meet  specific 
requirements.  Generating  various  number 
combinations  that  must  add  to  a  specified  sum. 

2.  Producing  a  number  of  more  involved 
relationships  among  a  few  symbols  to  satisfy 
general  requirements.  Generating  various 
operations  (within  a  domain,  such  as  calculus) 
that  could  be  used  to  reach  a  desired  result. 
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3.  Producing  a  number  of  complex  relationships 
among  a  few  ambiguous  symbols  to  satisfy 
ambiguous  requirements.  Generating  various  ways 
the  letters  JFK,  LBJ,  and  MLK  are  related. 

P.  DIVERGENT  PRODUCTION  OF  SYMBOLIC  SYSTEMS: 

1.  Generating  three  or  more  simple  symbols 
composing  a  collective  whole.  Examples  include 
telephone  or  Social  Security  numbers  or  letters 
forming  words. 

2 .  Generating  the  integration  of  many  moderately 
complex  symbols  to  meet  general  requirements. 
Words  organized  in  simple  sentences  or  symbols 
used  in  a  military  information  distribution 
system. 

3.  Generating  a  highly  complex  symbolic  system  to 
meet  ambiguous  requirements.  Developing  a  top 
secret  code  for  sending  highly  sensitive 
military  information. 

Q.  DIVERGENT  PRODUCTION  OF  SYMBOLIC 
TRANSFORMATIONS : 

1.  Generating  a  number  of  possible  changes  that  can 
be  imposed  on  a  simple  symbol  to  satisfy  a 
specific  requirement.  Producing  alternative 
printer  fonts  that  could  be  used  for  printing  a 
business  letter. 

2.  Generating  a  number  of  possible  changes  that  can 
be  imposed  on  a  moderately  complex  symbol  set. 
Producing  possible  updates  required  for  symbols 
on  a  map  display  given  the  status  of 

those  symbols  or  ways  of  measuring  out  exactly 
three  cups  of  water  given  a  five-cup  and  a 
two-cup  container. 

3.  Generating  a  number  of  possible  changes  in  a 
complex  symbol  set.  Producing  alternative  ways 
of  modifying  a  code  to  circumvent  attempts  to 
"break"  it. 

R.  DIVERGENT  PRODUCTION  OF  SYMBOLIC  IMPLICATIONS: 

1.  Generating  alternatives  on  the  nature  of 
information  contained  in  a  simple,  distinct, 
symbol.  Given  two  common  words,  produce  a 
variety  of  other  word  pairs  using  all  available 
letters  in  the  given  words. 

2.  Generating  alternatives  on  the  nature  of 
information  contained  in  possibly  ambiguous 
symbols.  Inferring  a  variety  of  equations  that 
follow  from  two  given  equations  composed  of 
letters. 
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3.  Generating  alternatives  on  the  nature  of 
information  contained  in  ambiguous,  complex 
symbols.  Given  a  swastika  and  the  Star  of 
David,  produce  a  variety  of  words  associated 
with  those  symbols. 

S.  DIVERGENT  PRODUCTION  OF  SEMANTIC  UNITS: 

1.  Generating  many  basic  ideas  for  a  given  simple 
concept.  Name  as  many  items  as  possible  that  are 
red  or  specify  things  that  fly. 

2.  Generating  many  ideas  for  a  given  moderately 
complex  concept.  Produce  as  many  ideas  as 
possible  that  are  green  and  inedible  or  common 
uses  for  a  rake. 

3.  Generating  many  ideas  for  a  given  a  complex 
concept.  Suggesting  titles  for  a  short  story  or 
producing  many  examples  of  things  that  are  warm, 
cuddly,  and  friendly. 

T.  DIVERGENT  PRODUCTION  OF  SEMANTIC  CLASSES: 

1.  Producing  many  classes  of  ideas  suggested  by  a 
given,  basic  concept.  Arranging  several  given 
words  in  a  variety  of  meaningful  groups. 

2.  Producing  many  classes  of  ideas  suggested  by  a 
given  moderately  complex  concept.  List 
alternative  uses  for  a  chair,  other  than  its 
common  use. 

3.  Producing  many  classes  of  ideas  suggested  by  a 
given  complex  concept.  Describe  a  variety  of 
uses  for  a  supercomputer  or  alternative  examples 
for  teaching  a  particular  concept. 

U.  DIVERGENT  PRODUCTION  OF  SEMANTIC  RELATIONS: 

1.  Producing  many  simple,  concrete  relationships 
among  a  few  concepts  or  situations.  Produce  a 
number  of  synonyms  for  a  given  word. 

2.  Producing  many  less  certain  relationships  among 
a  few  concepts.  Specify  a  number  of  words  to 
fill  a  blank  in  a  given  simile. 

3 .  Producing  many  complex  relationships  among 
potentially  ambiguous  concepts  or  ideas.  Specify 
ideas  related  to  the  concept  of  religion  or 
topics  related  to  the  theory  of  evolution. 

V.  DIVERGENT  PRODUCTION  OF  SEMANTIC  SYSTEMS: 

1.  Organizing  words  into  a  number  of  meaningful 
concepts  in  a  given,  concrete  context.  Writing 
some  alternative  headlines  for  a  major  news 
story  or  a  short  synopsis  of  a  specific  event. 

2.  Organizing  words  into  a  number  of  meaningful, 
moderately  complex  concepts  in  potentially 
ambiguous  contexts.  Given  five  single  letters, 
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composing  a  number  of  five-word  sentences  or 
completing  a  simile  giving  explanatory  remarks 
for  the  words  chosen. 

3.  Organizing  words  into  a  number  of  meaningful, 
complex  concepts  in  generally  ambiguous 
contexts.  Writing  an  article  about  a  new 
fashion  trend  or  producing  a  report  of  a 
patient's  condition. 

W.  DIVERGENT  PRODUCTION  OF  SEMANTIC 
TRANSFORMATIONS : 

1.  Producing  unconventional  (insightful) 
alternatives  based  on  a  reinterpretation  or  new 
emphasis  on  an  obvious  or  logical  aspect  of  an 
object  or  situation.  Producing  a  variety  of 
symbols  for  representing  given  activities  or 
objects. 

2.  Producing  unconventional  (insightful) 
alternatives  based  on  a  reinterpretation  or  new 
emphasis  on  a  moderately  ambiguous  aspect  of  an 
object  or  a  situation.  Producing  clever  or 
witty  titles  for  a  given  short  story. 

3.  Producing  unconventional  (insightful) 
alternatives  based  on  a  reinterpretation  or  new 
emphasis  on  some  aspect  of  an  object  or  a 
situation.  Solving  riddles  or  specifying  remote 
consequences  (based  on  distance  in  time,  space, 
or  probability)  of  a  given  event. 

X.  DIVERGENT  PRODUCTION  OF  SEMANTIC  IMPLICATIONS: 

1.  Generating  many  possible  causes,  parallel 
events,  or  outcomes  suggested  by  a  simple 
concept.  Given  a  light  bulb  does  not  work, 
specify  several  possible  causes  and  several 
possible  consequences. 

2.  Generating  many  possible  causes,  parallel 
events,  or  outcomes  suggested  by  a  moderately 
complex  concept.  Given  a  tape  recorder  does  not 
work,  specify  several  possible  causes  and 
several  possible  consequences. 

3.  Generating  many  possible  causes,  parallel 
events,  or  outcomes  suggested  by  a  complex 
concept.  Given  a  heart-lung  machine  is 
inoperative,  specify  several  possible  causes  and 
several  possible  consequences. 

Y.  DIVERGENT  PRODUCTION  OF  BEHAVIORAL  UNITS: 

1.  Generating  a  number  of  simple,  distinct, 

behavioral  alternatives  to  satisfy  a  specific 
situation  or  condition.  Given  the  situation  of  a 
birthday  party,  supposing  that  smiling  laughing 
and  talking  will  be  appropriate  behavior. 


2.  Generating  a  number  of  complex  behavioral 
alternatives  to  satisfy  a  moderately  ambiguous 
situation  or  condition.  Given  the  situation  of  a 
wedding,  supposing  that  making  conversation 
dancing  with  the  bride  or  groom,  and  toasting 
the  couple  will  all  be  appropriate  activities. 

3.  Generating  a  number  of  complex  behavioral 
alternatives  to  satisfy  an  ambiguous  situation 
or  condition.  Given  the  situation  of  being  held 
hostage  by  terrorists,  supposing  that  lying, 
stealing,  and  even  killing  may  be  appropriate  to 
insure  survival. 

Z.  DIVERGENT  PRODUCTION  OF  BEHAVIORAL  CLASSES: 

1.  Producing  various  alternative  ways  of 
classifying  behavioral  items  by  a  few  simple  or 
readily  apparent  attributes.  Group  simple 
actions  (smiling,  waving,  eating,  etc.)  in  a 
variety  of  ways. 

2.  Producing  alternative  ways  of  classifying 
moderately  complex  behavioral  activities  on 
several  potentially  ambiguous  or  not  readily 
apparent  dimensions.  Group  funerals,  birthdays, 
and  anniversaries  on  a  number  of  meaningful 
dimensions. 

3.  Producing  various  alternative  ways  of 
classifying  complex  behavioral  activities  on 
several  potentially  ambiguous  or  not  readily 
apparent  dimensions.  Group  the  complex  human 
activities  of  war,  politics,  literary  pursuit, 
and  exploration  in  several  different  ways. 

AA.  DIVERGENT  PRODUCTION  OF  BEHAVIORAL  RELATIONS: 

1.  Producing  a  number  of  simple  relationships  among 
a  few  distinct  behavioral  actions  or  situations. 
Specify  how  a  frown,  a  grimace  and  smile  might 
be  related. 

2.  Producing  a  number  of  moderately  complex 
relationships  among  potentially  ambiguous 
behavioral  actions  or  situations.  Specify  how  a 
judge,  a  policeman,  and  a  tax  collector  might  be 
related  in  terms  of  their  professions. 

3 .  Producing  a  number  of  complex  relationships 
among  potentially  ambiguous  behavioral  actions 
or  situations.  Specify  how  any  two  given  members 
of  the  European  Community  are  economically  and 
politically  related. 

BB.  DIVERGENT  PRODUCTION  OF  BEHAVIORAL  SYSTEMS: 

1.  Generating  the  alternative  interactions  and 
acceptable  actions  in  a  simple,  well-defined, 
social  system.  Given  the  context  of  routine 
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military  operations,  produce  appropriate 
behaviors  for  an  enlisted  man  encountering  an 
officer. 

Generating  the  alternative  interactions  and 
acceptable  actions  in  a  moderately  complex 
social  system.  Given  the  context  of  a  busy 
street  intersection,  produce  appropriate 
behaviors  for  a  young  man  encountering  an 
elderly  woman. 

Generating  the  alternative  interactions  and 
acceptable  actions  in  a  complex  social  system 
with  varying  degrees  of  ambiguity.  Given  the 
context  of  chaos  following  a  natural  disaster, 
produce  appropriate  behaviors  for  survivors. 

DIVERGENT  PRODUCTION  OF  BEHAVIORAL 
TRANSFORMATIONS : 

Producing  unconventional  (insightful) 
alternatives  based  on  a  reinterpretation  or  new 
emphasis  on  a  well-defined  object  or  situation. 

A  basketball  coach  generating  possible 
modifications  to  an  existing  offense  for 
responding  (exploiting)  a  particular  defensive 
strategy  employed  by  an  opposing  team. 

Producing  unconventional  (insightful) 
alternatives  based  on  a  reinterpretation  or  new 
emphasis  on  a  moderately  complex  object  or 
situation.  A  company  Chief  Executive  Officer 
developing  alternative  business  strategies  and 
policies  to  insure  the  company's  future 
competitiveness,  based  on  hazy  financial  and 
economic  forecasts. 

Producing  unconventional  (insightful) 
alternatives  based  on  a  reinterpretation  or  new 
emphasis  on  a  complex  object  or  situation. 

Choreographing  a  ballet  to  portray  changing  mood 
conveyed  by  the  music  or  storyline. 

DIVERGENT  PRODUCTION  OF  BEHAVIORAL  IMPLICATIONS: 

Generating  many  possible  causes,  parallel 
events,  or  outcomes  suggested  by  a  simple 
behavioral  concept.  Given  a  smiling  person, 
specify  several  possible  causes  of  the  smile  and 
several  possible  consequences. 

Generating  many  possible  causes,  parallel 
events,  or  outcomes  suggested  by  a  moderately 
complex  behavioral  concept.  Given  a  person  who 
has  just  received  a  speeding  ticket,  specify 
several  possible  behavioral  consequences  and 
several  ways  to  ameliorate  any  negative 
consequences. 
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3.  Generating  many  possible  causes,  parallel 
events,  or  outcomes  suggested  by  a  complex 
concept.  Given  a  depressed  person,  specify 
several  possible  causes,  several  possible 
consequences,  and  several  recommended 
treatments . 
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SECTION  4.  CONVERGENT  PRODUCTION.  Retrieving  from  memory 
store  a  specific  element  of  information,  which  involves  the 
generation  of  logical  conclusions  based  on  given  information. 
Traditionally  identified  as  deductive  reasoning.  Proceeding 
to  an  answer  by  following  rules  or  algorithms,  such  as 
determining  a  friend's  age  by  subtracting  his  birthyear  from 
the  current  year,  or  determining  the  question  to  a  Jeopardy! 
answer. 

YES _  (answer  remaining  questions)  NO _  (proceed  to 

next  section) 

F  -  C 

A.  CONVERGENT  PRODUCTION  OF  VISUAL  UNITS: 

_  -  _  1.  Producing  a  simple,  distinct,  visual  form  to 

satisfy  a  specific  requirement.  Drawing  a 
square  when  asked  or  making  a  simple  picture  for 
a  specific  item. 

_  -  _  2 .  Producing  a  moderately  complex  visual  form  to 

satisfy  a  specific  requirement.  Drawing  a  car 
with  four  wheels,  windshield,  and  other  gross 
descriptive  elements  when  asked  to  produce  an 
automobile. 

_  -  _  3.  Producing  a  complex  visual  form  to  satisfy  a 

specific  requirement.  Drawing  a  highly  detailed 
image  of  a  microwave  oven  (showing 
digital  readout,  knobs,  logos,  etc.)  when  asked 
to  produce  a  modern  kitchen  appliance  designed 
to  cook  food  quickly. 

F  -  C 

B.  CONVERGENT  PRODUCTION  OF  VISUAL  CLASSES: 

_  -  _  1.  Determining  the  classification  of  a  non-complex 

visual  item  by  a  specified,  well-defined, 
attribute.  Sort  the  letters  of  the  alphabet  as 
either  containing  curved  lines  or  not  or  given 
five  polygons  select  all  with  four  or  less 
sides. 

_  -  _  2.  Determining  the  classification  of  a  moderately 

complex  visual  item  based  on  a  specified 
attribute.  Given  15  bicycles,  classify  each 
them  in  two  groups,  one  of  imports,  the  other  of 
domestic  brands  or  grade  fresh  fruit  for  various 
standards  (i.e.,  Grade  A  or  Extra  Fancy). 

_  -  _  3.  Determining  the  classification  of  a  complex 

visual  item  by  a  specified  attribute.  Sort  10 
patients  into  three  groups,  based  on  the  level 
of  medical  attention  required,  based  on  the 
patient's  physical  appearance. 

F  -  C 

C.  CONVERGENT  PRODUCTION  OF  VISUAL  RELATIONS: 

_  -  _  1.  Specifying  simple  relationships  among  a  few 

distinct  visual  forms  to  satisfy  a  specific 
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requirement.  Given  five  polygons,  specify  which 
(if  any)  are  related  based  on  the  number  of 
sides . 

2.  Specifying  more  involved  relationships  among  a 
few  moderately  complex  visual  forms  to  satisfy  a 
specific  requirement.  Given  six  photographs  of 
single-family  dwellings,  specify  which  (if  any) 
are  related  based  on  type  of  construction  (i.e., 
brick,  frame,  stucco,  etc.). 

3.  Specifying  complex  relationships  among  a  few 
complex  visual  forms  to  satisfy  a  specific 
requirement.  Given  four  aerial  reconnaissance 
photographs  of  a  suspected  missile  battery, 
specify  the  relationships  of  the  various  objects 
in  the  photographs  to  one  another. 

D.  CONVERGENT  PRODUCTION  OF  VISUAL  SYSTEMS: 

1.  Producing  the  static  arrangement  and  position  a 
few  simple  visual  forms  to  satisfy  a  specific 
requirement.  Specifying  the  optimum  arrangement 
of  appliances  in  a  kitchen  based  on  the 
requirement  of  minimizing  the  distance  between 
the  appliances. 

2.  Producing  the  arrangement  and  spatial  position 
of  several  moderately  complex  objects  to  satisfy 
a  specific  requirement.  Developing 

the  optimum  layout  of  machines  and  processes  in 
a  factory  to  achieve  the  greatest  rate  of 
production  per  operating  hour. 

3.  Producing  the  arrangement  and  spatial  positions 
of  members  of  a  complex  system  of  visual  forms 
to  satisfy  a  specific  requirement.  Directing 
the  altitude  and  headings  for  all  aircraft 
waiting  to  land  at  a  busy  airport  to  insure  the 
safety  of  each  aircraft. 

E.  CONVERGENT  PRODUCTION  OF  VISUAL  TRANSFORMATIONS: 

1.  Decomposing  a  simple  image  and  reconstructing  it 
to  conform  to  a  specific  requirement.  Given  12 
match  sticks  forming  a  square  divided  into 
quadrants  (thereby  creating  five  squares)  remove 
any  two  matches  to  leave  three  squares. 

2.  Decomposing  a  moderately  complex  image  and 
reconstructing  it  to  conform  to  a  specific 
requirement.  Revising  a  landscaping  plan  to 
incorporate  more  shade  tree  and  fewer  shrubs  or 
specifying  a  new  exterior  body  shape  to 
accommodate  a  requirement  for  a  lower 
coefficient  of  drag  in  an  automobile  design. 

3.  Decomposing  a  complex  image  and  reconstructing 
it  to  conform  to  a  specific  requirement. 
Attempting  to  identify  enemy  tanks  and 
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emplacements  that  are  camouflaged  or 
reconstructing  a  victim's  facial  features  based 
on  skeletal  remains. 

F.  CONVERGENT  PRODUCTION  OF  VISUAL  IMPLICATIONS: 

1.  Determining  the  information  inherent  in  a  simple 
visual  scene.  Following  the  directions  of  a 
simple  sign  (i.e.,  a  policeman's  extended  arm 
implying  "STOP") . 

2 .  Determining  the  information  inherent  in  a 
moderately  complex  visual  scene.  Seeing  an  open 
window  with  rain  blowing  in  and  closing  the 
window  or  seeing  smoke  coming  out  of  a  house  and 
calling  the  fire  department. 

3.  Determining  the  information  inherent  in  a  simple 
visual  scene.  Anticipating  an  erratic  driver's 
swerving  to  avoid  a  head-on  collision  or 
determining  the  nature  of  patient's  illness, 
based  on  a  visual  inspection  of  the  patient 

G.  CONVERGENT  PRODUCTION  OF  AUDITORY  UNITS: 

1.  Producing  a  simple,  distinct,  auditory  stimulus 
to  satisfy  a  specific  requirement. 

Barking  when  asked  how  a  dog  sounds  or  playing  a 
specific  musical  note. 

2.  Producing  a  moderately  complex  auditory  stimulus 
to  satisfy  a  specific  requirement.  Playing  a 
specific  chord  on  an  instrument. 

3.  Producing  a  complex  auditory  stimulus  to  satisfy 
a  specific  requirement.  Giving  the  correct 
pronunciation  of  a  word. 

H.  CONVERGENT  PRODUCTION  OF  AUDITORY  CLASSES: 

I.  Determining  the  classification  of  a  simple 
auditory  stimulus  based  on  a  specified 
attribute.  Given  a  tone,  specify  if  it  belongs 
in  one  of  several  categories. 

2.  Determining  the  classification  of  a  moderately 
complex  auditory  stimulus  based  on  a  specified 
attribute.  Given  a  piece  of  music,  specify  its 
general  class  (i.e.,  classical,  pop,  rock, 
gospel,  etc.). 

3.  Determining  the  classification  of  a  complex 
auditory  stimulus  based  on  a  specified 
attribute.  Identifying  a  person  by  their  voice 
or  a  specific  animal  by  its  call. 

I.  CONVERGENT  PRODUCTION  OF  AUDITORY  RELATIONS: 

l.  Specifying  simple  relationships  among  a  few 
distinct  auditory  stimuli  to  satisfy  a  specific 
requirement.  Given  five  musical  notes,  specify 
which  (if  any)  are  related  based  on  their  key. 
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2.  Specifying  more  involved  relationships  among  a 
few  auditory  stimuli  to  satisfy  a  specific 
requirement.  Given  six  engine  noises,  specify 
which,  (if  any)  is  related  to  low  oil  pressure. 

3.  Specifying  complex  relationships  among  a  few 
complex  auditory  stimuli  to  satisfy  a  specific 
requirement.  Given  four  operettas,  specify  which 
(if  any)  are  related  based  on  the  style  of 
musical  composition  or  if  any  of  three  voices 
have  the  same  regional  accent. 

J.  CONVERGENT  PRODUCTION  OF  AUDITORY  SYSTEMS: 

1.  Producing  the  arrangement  and/or  position  of  a 
few  simple  auditory  stimuli  to  satisfy  a 
specific  requirement.  Identifying  a  bugler's 
signal  to  charge  or  a  simple  drum  cadence. 

2.  Producing  the  arrangement  of  several  moderately 
complex  auditory  stimuli  to  satisfy  a  specific 
requirement.  Identifying  some  of  the  principle 
sounds  in  a  busy  bus  station  or  interpreting  an 
expected  message  on  the  telephone. 

3.  Producing  the  arrangement  and/or  position  of  a 
complex  system  of  auditory  stimuli  to  satisfy  a 
specific  requirement.  Identifying  a  radio 
communication  that  is  partially  jammed  or 
identifying  the  position  of  a  submarine  from 
several  auditory  inputs. 

K.  CONVERGENT  PRODUCTION  OF  AUDITORY 
TRANSFORMATIONS : 

1.  Decomposing  a  simple  sound  and  reconstructing  it 
to  conform  to  a  specific  requirement.  Producing 
the  result  of  increasing  a  700  Hz  tone  by  200 
Hz. 

2.  Decomposing  a  moderately  complex  sound  and 
reconstructing  it  to  conform  to  a  specific 
requirement.  Producing  the  result  of  changing 
from  a  D-minor  to  a  D-major  chord. 

3.  Decomposing  a  complex  sound  and  reconstructing 
it  to  conform  to  a  specific  requirement.  An 
impressionist  changing  his  voice  to  sound  like  a 
famous  actor. 

L.  CONVERGENT  PRODUCTION  OF  AUDITORY  IMPLICATIONS: 

1.  Determining  the  information  inherent  in  a  simple 

auditory  stimulus.  Following  the  directions  of 
a  simple  sound  (i.e.,  stopping  at  a  railroad 
crossing  after  hearing  the  warning  bell) . 
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2.  Determining  the  information  inherent  in  a 
moderately  complex  auditory  stimulus.  Hearing  a 
sputtering  lawn-mower  engine  and  knowing  it 
needs  a  tune-up. 

3.  Determining  the  information  inherent  in  a 
complex  auditory  stimulus.  Determining  the 
emotions  implied  in  a  moving  piece  of  music  or 
the  seriousness  of  a  patient's  lung  ailment 
based  on  the  sound  of  his  breathing. 

M.  CONVERGENT  PRODUCTION  OF  SYMBOLIC  UNITS: 

1.  Producing  a  simple,  distinct,  symbol  to  satisfy 
a  specific  requirement.  Producing  a  plus  (+) 
sign  when  asked  for  the  operator  depicting 
addition. 

2.  Producing  a  moderately  complex  symbol  to  satisfy 
a  specific  requirement.  Producing  the  symbol 
representing  Chrysler  Corporation  when  asked  for 
its  company  logo. 

3.  Producing  a  complex  symbol  to  satisfy  a  specific 
requirement.  Producing  the  Declaration  of 
Independence  when  asked  for  the  symbol  of  the 
United  States'  formal  break  with  England. 

N.  CONVERGENT  PRODUCTION  OF  SYMBOLIC  CLASSES: 

1.  Determining  the  classification  of  a  simple 
symbol  based  on  a  specified  attribute.  Given  a 
pound  sign  (#),  specify  if  it  can  be  used  as  a 
mathematic  operator. 

2.  Determining  the  classification  of  a  simple 
symbol  based  on  a  specified  attribute.  Given  a 
company  logo,  specify  its  general  industrial 
class  (i.e.,  steel,  aerospace,  pharmaceuticals, 
etc.)  or  given  a  national  flag  identify  the  name 
of  its  country. 

3.  Determining  the  classification  of  a  simple 
symbol  based  on  a  specified  attribute.  Given 
the  name  of  a  place  (i.e.,  Bunker  Hill) 
determine  if  the  place  is  of  patriotic 
significance. 

O.  CONVERGENT  PRODUCTION  OF  SYMBOLIC  RELATIONS: 

1.  Completing  a  simple  symbolic  relationship.  Given 
a  sequence  of  three  numbers  (1,  3,  5)  specify 
which  number  logically  follows  (7) . 

2.  Completing  a  more  involved  symbolic 
relationship.  Breaking  a  simple  code  or  applying 
rules  for  spelling  words. 

3.  Completing  a  complex  symbolic  relationship. 

Given  a  brief  line  or  term  (i.e.,  "separate  but 
equal"  or  "I  have  a  dream")  identify  the 
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relationship  of  those  terms  or  deciphering  a 
complex  code. 


P.  CONVERGENT  PRODUCTION  OF  SYMBOLIC  SYSTEMS: 

1.  Producing  a  simple,  completely  structured  order 
of  symbols  to  satisfy  a  specific  requirement. 
Developing  a  simple  identification  system  such 
as  alphabetizing  names  on  a  list. 

2.  Producing  a  moderately  complex,  completely 
structured  order  of  symbols  to  satisfy  a 
specific  requirement.  Writing  a  crossword  puzzle 
given  a  set  of  words  or  developing  a  ZIP  code 
system. 

3.  Producing  a  complex,  completely  structured  order 
of  symbols  to  satisfy  a  specific  requirement. 
Designing  a  complex  code  for  transmitting 
sensitive  or  top-secret  information  of  national 
importance . 

Q.  CONVERGENT  PRODUCTION  OF  SYMBOLIC 
TRANSFORMATIONS : 

1.  Producing  new  symbolic  elements  of  information 
by  revising  a  given  basic  symbol  to  meet  a 
specified  requirement.  Finding  the  name  of  a 
given  object  (i.e.,  ball)  hidden  in  a  sentence 
(i.e.,  "We  saw  the  dancer  throw  a  baton  at  the 
BALLet" ) . 

2.  Producing  new  symbolic  elements  of  information 
by  revising  a  given,  moderately  complex  symbol 
to  meet  a  specified  requirement.  Updating  a 
given  symbol  (i.e.,  a  green  aircraft)  on  a  map 
to  reflect  its  current  status  (changing  the 
color  from  green  to  yellow  to  indicate  low 
fuel) . 

3.  Producing  new  symbolic  elements  of  information 
by  revising  a  given  complex  symbol  to  meet  a 
specified  requirement.  Producing  a  new  symbol 
(i.e.,  company  logo)  to  reflect  changes  in  the 
company. 

R.  CONVERGENT  PRODUCTION  OF  SYMBOLIC  IMPLICATIONS: 

1.  Producing  a  unambiguous  symbolic  deduction  from 
given  basic  symbolic  information  in  a  novel  or 
unpracticed  context.  Applying  mathematics  to 
develop  a  personal  budget  (given  that  one  has 
never  used  a  budget)  or  using  blueprint  plans  to 
construct  a  simple  piece  of  furniture. 

2.  Producing  a  unambiguous  symbolic  deduction  from 
given  moderately  complex  symbolic  information  in 
a  novel  or  unpracticed  context.  Applying 
calculus  to  solve  an  engineering  problem  or 
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sight-reading  a  piece  of  music  for  the  first 
time . 

3.  Producing  a  unambiguous  symbolic  deduction  from 
given  complex  symbolic  information  in  a  novel  or 
unpracticed  context.  Determining  the  correct 
surgical  approach  for  a  previously  unencountered 
injury  or  applying  quantum  theory  to  predict  the 
behavior  of  particles  in  a  particular  chemical 
reaction. 

S.  CONVERGENT  PRODUCTION  OF  SEMANTIC  UNITS: 

1.  Producing  an  appropriate  object  or  idea  to 
summarize  a  given  basic  concept.  Answering  a 
crossword  puzzle  clue  (i.e.,  a  five-letter  word 
for  "noontime  meal”  -  lunch)  or  specifying  the 
proper  tool  in  a  basic  carpentry  job. 

2.  Producing  an  appropriate  object  or  idea  to 
summarize  a  given  moderately  complex  concept. 
Indicating  a  micro-computer  system  that  is 
suitable  for  a  particular  business  operation  or 
selecting  a  house  that  meets  the  needs  of  a 
family. 

3.  Producing  an  appropriate  object  or  idea  to 
summarize  a  given  complex  concept.  Specifying  a 
word  when  asked  to  describe  someone  with  "just 
one  word"  or  an  evaluator  assigning  a  single 
rating  to  a  multi-billion  dollar  bid  proposal 
for  a  new  superhighway  (i.e.,  "acceptable"  or 
"unacceptable) . 

T.  CONVERGENT  PRODUCTION  OF  SEMANTIC  CLASSES: 

1.  Producing  meaningful  classes  of  simple  objects 
or  concepts  to  satisfy  a  specific  condition(s) . 
Grouping  nine  given  words  in  only  three 
categories,  with  no  words  left  over  or 
specifying  the  general  class  of  a  group  of 
objects. 

2.  Producing  meaningful  classes  of  moderately 
complex  objects  or  concepts  to  satisfy  a 
specific  condition(s) .  Given  a  group  of  four 
items,  divide  then  into  three  more  specific 
classes. 

3.  Producing  meaningful  classes  of  complex  objects 
or  concepts  to  satisfy  a  specific  condition (s) . 
Given  the  same  four  items  as  above,  specify  an 
even  more  precise  class  (such  as  the  model  and 
year  for  automobiles) . 

U.  CONVERGENT  PRODUCTION  OF  SEMANTIC  RELATIONS: 

1.  Specifying  an  idea  or  concept  that  satisfies  a 
given,  simple,  relationship.  Pronounce  the 
antonym  of  a  given  word  given  the  first  letter 
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or  determine  a  word  similar  in  meaning  to  two 
other  words. 

2.  Specifying  an  idea  or  concept  that  satisfies  a 
given,  moderately  complex  relationship.  Given 
the  words  "father, son,  mother,"  determine  the 
completing  word. 

3.  Specifying  an  idea  or  concept  that  satisfies  a 
given,  complex,  relationship.  Developing  a 
theory  to  explain  complex  physical  phenomenon 
(i.e.,  the  theory  of  relativity  was  developed  to 
explain  the  relationship  of  matter  and  energy) . 

V.  CONVERGENT  PRODUCTION  OF  SEMANTIC  SYSTEMS: 

1.  Integrating  three  or  more  concepts  in  a 
meaningful,  simple,  sequence.  Developing  the 
rules  of  a  simple  game  or  creating  a  simple 
sentence  to  convey  a  specific  meaning  (i.e., 
"Close  the  door") . 

2.  Integrating  three  or  more  concepts  in  a 
meaningful,  moderately  complex,  sequence.  Given 
five  scrambled  pictures  from  a  comic,  indicate 
the  order  that  makes  sense  or  order  several 
given  sentences  in  a  meaningful  way. 

3.  Integrating  three  or  more  concepts  in  a 
meaningful,  complex,  sequence.  Specify  the 
temporal  sequence  of  steps  needed  to  execute  a 
complex  plan  (i.e.,  manufacturing  a  new 
product) . 

W.  CONVERGENT  PRODUCTION  OF  SEMANTIC 
TRANSFORMATIONS : 

1.  Determining  novel  applications  of  simple  objects 
and  concepts  by  removing  them  from  their 
traditional  contexts  and  redefining  them  to 
satisfy  a  specified  purpose.  Given  three  simple 
objects  (a  broom,  a  box,  and  a  plate)  choose  one 
to  retrieve  a  kite  caught  in  a  small  tree. 

2.  Determining  novel  applications  of  moderately 
complex  objects  and  concepts  by  removing  them 
from  their  traditional  contexts  and  redefining 
them  to  satisfy  a  specified  purpose.  Placing 
four-wheel  drive  transmissions  in  passenger  cars 
to  increase  traction  and  versatility  or  adapting 
a  standard  course  to  a  new  clientele. 

3.  Determining  novel  applications  of  complex 
objects  and  concepts  by  removing  them  from  their 
traditional  contexts  and  redefining  them  to 
satisfy  a  specified  purpose.  Synthesizing  a  new 
product  from  two  existing  products  (i.e., 
combining  the  power  of  the  computer  and  laser 
printers  to  create  "desktop  publishing")  or 


combining  the  qualities  of  spacecraft  and 
aircraft  to  create  the  Space  Shuttle. 

X.  CONVERGENT  PRODUCTION  OF  SEMANTIC  IMPLICATIONS: 

1.  Deducing  meaningful  information  inherent  in  a 
given  simple  concept  or  situation.  Given  a 
specific  event  and  context,  specify  the 
appropriate  event  which  immediately  follows. 

2.  Deducing  meaningful  information  inherent  in 
given  moderately  complex  concept  or  situation. 
While  flying,  the  pilot  notices  his  number  two 
engine  has  failed;  he  then  executes  emergency 
procedures  to  restart  the  engine. 

3.  Deducing  meaningful  information  inherent  in  a 
given  complex  concept  or  situation.  Reading  a 
book  and  appreciating  its  themes  or  military 
intelligence  officers  examining  many  sources  of 
data  on  enemy  troop  movements  and  exercises  to 
determine  their  readiness  and  intentions. 

Y.  CONVERGENT  PRODUCTION  OF  BEHAVIORAL  UNITS: 

1.  Producing  a  simple,  distinct,  behavioral  action 
to  satisfy  a  given  requirement. 

Determining  the  appropriate  response  in  an 
unambiguous  social  situation,  such  as  shaking 
hands  when  you  are  introduced  to  someone  who 
offers  their  hand. 

2.  Producing  a  moderately  complex  behavioral  action 
to  satisfy  a  given  requirement.  Making  a  toast 
at  a  wedding  reception  when  asked  to  do  so. 

3 .  Producing  a  complex  behavioral  action  to  satisfy 
a  given  requirement.  As  an  actress,  portraying 
a  woman  who  has  just  received  news  that  her 
husband  has  been  killed  in  action  during 
wartime . 

Z.  CONVERGENT  PRODUCTION  OF  BEHAVIORAL  CLASSES: 

1.  Producing  meaningful  classes  of  simple  behaviors 
to  satisfy  a  specific  condition (s) .  Grouping 
five  basic  behaviors  in  only  two  categories  or 
specifying  the  general  class  of  a  group  of 
behaviors . 

2.  Producing  meaningful  classes  of  moderately 
complex  behaviors  to  satisfy  a  specific 
condition (s) .  Given  a  group  of  six  social 
events,  divide  them  into  three  specific  classes, 
placing  each  in  only  one  class  (i.e.,  family 
events,  office  events,  and  political  events) . 

3 .  Producing  meaningful  classes  of  complex 
behaviors  to  satisfy  a  specific  condition(s) . 
Given  five  different  wars,  classify  each  as 
either  a  limited-war  or  an  all-out  war. 
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AA.  CONVERGENT  PRODUCTION  OF  BEHAVIORAL  RELATIONS: 

1.  Specifying  a  behavior  appropriate  to  a  given, 
concrete,  relationship.  A  butler  determining 
the  correct  manner  to  greet  his  employer. 

2.  Specifying  a  behavior  appropriate  to  a  given, 
moderately  complex,  relationship.  A  manager 
determining  the  correct  manner  to  greet  a 
colleague  from  another  division  of  the  same 
company. 

3.  Specifying  a  behavior  appropriate  to  a  given, 
complex,  relationship.  A  head  of  state  (and  her 
staff)  determining  the  proper  manner  of  greeting 
a  visiting  head  of  state  from  a  rival  nation. 

BB.  CONVERGENT  PRODUCTION  OF  BEHAVIORAL  SYSTEMS: 

1.  Integrating  three  or  more  behaviors  in  a 
meaningful,  simple,  sequence  to  achieve  a 
specified  goal.  Deciding  the  actions  required 
to  complete  a  simple  task  (i.e.,  the  steps 
needed  to  clean  the  house) . 

2.  Integrating  three  or  more  behaviors  in  a 
meaningful,  moderately  complex,  sequence  to 
achieve  a  specified  goal.  Planning  how  a  stage 
crew  will  change  the  scenery  between  acts  of  a 
play  or  developing  the  schedule  for  a  national 
conference  of  a  professional  society. 

3 .  Integrating  three  or  more  behaviors  in  a 
meaningful,  complex,  sequence  to  achieve  a 
specified  goal.  Coordinating  a  major  military 
maneuver  (i.e.,  the  D-Day  Invasion)  or 
determining  the  necessary  movements  to  catch  a 
thrown  ball. 

CC.  CONVERGENT  PRODUCTION  OF  BEHAVIORAL 
TRANSFORMATIONS : 

1.  Determining  novel  applications  of  simple 
behaviors  by  removing  them  from  their 
traditional  contexts  and  redefining  them  to 
satisfy  a  specified  purpose.  When  building  a 
shelf,  a  carpenter  notices  that  a  modification 
of  a  joint  he  uses  in  securing  ceiling  beams  can 
be  used  to  join  shelf  supports. 

2.  Determining  novel  applications  of  moderately 
complex  behaviors  by  removing  them  from  their 
traditional  contexts  and  redefining  them  to 
satisfy  a  specified  purpose.  Determining  how  a 
previously  successful  problem  solving  approach 
can  be  modified  to  solve  a  current  problem. 

3.  Determining  novel  applications  of  complex 
behaviors  by  removing  them  from  their 
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traditional  contexts  and  redefining  them  to 
satisfy  a  specified  purpose.  A  psychiatrist 
tailoring  a  standard  therapeutic  technique  for 
the  unique  needs  of  his  patient. 

DD.  CONVERGENT  PRODUCTION  OF  BEHAVIORAL 
IMPLICATIONS: 

1.  Deducing  meaningful  information  inherent  in  a 
given  simple  behavioral  situation.  Determining 
that  a  person  is  sad  because  he  is  sobbing  or 
that  an  audience  approves  of  an  entertainer's 
performance  given  a  standing  ovation  is 
observed. 

2.  Deducing  meaningful  information  inherent  in 
given  moderately  complex  behavioral  situation. 

As  a  defensive  player,  attempting  to  determine 
the  real  intent  of  opposing  players  while  they 
execute  a  "decoy"  play  designed  to  fool  the 
defense. 

3.  Deducing  meaningful  information  inherent  in  a 
given  complex  behavioral  situation.  Determining 
a  child  has  been  emotionally  abused  based  on  his 
actions  and  answers  to  diagnostic  test  questions 
or  determining  the  mood  of  stockholders 
regarding  an  attempted  hostile  takeover  of  the 
company. 
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SECTION  5.  EVALUATION.  This  operation  involves  reaching 
decisions  or  making  assessments  relative  to  some  criterion. 
This  can  include  judgements  regarding  correctness,  identity, 
adequacy,  consistency,  desirability,  or  other  attributes  of 
information.  Examples  of  this  operation  include  determining 
if  a  proposed  highway  will  interfere  with  animal  migration 
patterns,  if  a  glass  will  hold  the  entire  contents  of  a 
beverage  container,  and  if  a  Macintosh  apple  belongs  to  the 
class  of  firm,  red,  edible  objects. 

YES _  (answer  remaining  questions)  NO _  (stop) 

F  -  C 

A.  EVALUATION  OF  VISUAL  UNITS: 

_  -  _  1.  Judging  a  simple,  distinct,  visual  form  as  being 

similar  or  different  to  each  other  or  a 
specified  standard.  Given  two  shapes,  evaluate 
their  similarity  or  compare  a  given  shape  to  a 
target  shape. 

_  -  _  2 .  Judging  a  moderately  complex  visual  form  as 

being  similar  or  different  to  each  other  or  a 
specified  standard.  Given  two  kitchen  utensils, 
evaluate  their  similarity  or  compare  a  given 
appliance  to  a  target  appliance. 

_  -  _  3.  Judging  a  complex  visual  form  as  being  similar 

or  different  to  each  other  or  a  specified 
standard.  Given  two  automobiles,  evaluate  their 
similarity  or  compare  a  given  style  of 
architecture  to  a  desired  (target)  style  of 
architecture. 

F  -  C 

B.  EVALUATION  OF  VISUAL  CLASSES: 

_  -  _  1.  Evaluating  the  classification  of  a  non-complex 

visual  item  based  on  a  specified,  well-defined, 
attribute.  Given  a  group  of  five  cans  with 
crooked  labels  (marked  "failed")  determine  the 
classification  rule  and  apply  it  to  evaluate 
other  cans. 

_  -  _  2.  Evaluating  the  classification  of  a  moderately 

complex  visual  item  based  on  a  specified 
attribute.  Given  a  stack  of  lumber  for  interior 
woodwork  (marked  "top  grade")  determine  the 
classification  rule  based  on  grain,  lack  of 
knots,  finish,  straightness,  etc.,  and  apply  it 
to  grade  other  stacks  of  lumber. 

_  -  _  3.  Evaluating  the  classification  of  a  complex 

visual  item  based  on  a  specified  attribute. 

Given  several  examples  of  "great  art 
masterpieces,"  determine  the  classification  rule 
and  apply  it  to  other  works  of  art. 
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C.  EVALUATION  OF  VISUAL  RELATIONS: 

1.  Judging  given,  simple  relationships  among  a  few 
distinct  visual  forms,  relative  to  a  specified 
standard.  Evaluating  the  placement  of  plates  and 
utensils  at  a  table  to  serve  five  people  at 
dinner  or  arrangement  of  books  on  a  shelf  from 
taller  books  on  the  left  to  shorter  books  to  the 
right. 

2.  Judging  given,  more  involved  relationships  among 
a  few  visual  forms,  relative  to  a  specified 
standard.  Evaluating  the  symmetry  of  pictures 
hung  on  a  wall  or  the  relationship  of  frequently 
used  controls  to  each  other  on  an  automobile 
dashboard. 

3.  Judging  given,  complex  relationships  among  a  few 
potentially  ambiguous  visual  forms  and 
dimensions,  relative  to  a  specified  standard. 
Evaluating  the  suitability  of  a  given  color 
scheme  for  decorating  a  new  living  room  to 
convey  a  desired  mood. 

D.  EVALUATION  OF  VISUAL  SYSTEMS: 

1.  Analyzing  the  static  arrangement  and  position  a 
few  simple  visual  forms  conforming  to  a 
specified  requirement.  Evaluating  a  given 
arrangement  of  appliances  in  a  kitchen  designed 
to  reduce  the  workload  on  the  cook. 

2.  Analyzing  the  arrangement  and  spatial  position 
of  several  moderately  complex  objects  conforming 
to  a  specified  requirement.  Appraising  the 
suitability  of  a  houseplan  to  accommodate  a 
family's  specific  needs. 

3.  Analyzing  the  arrangement  and  spatial  positions 
of  members  of  a  complex  system  of  visual  forms 
conforming  to  a  specified  requirement. 

Appraising  the  positions  of  all  aircraft  in  a 
holding  pattern  over  a  busy  airport  to  insure 
adequate  spacing  for  each  aircraft. 

E.  EVALUATION  OF  VISUAL  TRANSFORMATIONS: 

1.  Appraising  the  impact  of  some  specific  change  on 
a  simple  visual  object.  Judging  that  a  block  or 
some  other  simple  object  has  rotated  270  degrees 
or  than  an  object  on  a  desk  has  been  moved. 

2.  Appraising  the  impact  of  several  changes  in  a 
moderately  complex  visual  scene  or  object. 
Judging  if  several  different  perspective  views 
are  of  the  same  boat. 
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3.  Appraising  the  impact  of  subtle  changes  in  a 
complex  visual  scene  or  object.  Judging  the 
cause  of  an  aircraft  accident  based  on  visual 
inspection  of  the  crash  site  and  wreckage. 

F.  EVALUATION  OF  VISUAL  IMPLICATIONS: 

1.  Judging  the  consequences  inherent  in  a  simple 
visual  scene  given  a  specific  standard. 
Evaluating  if  a  particular  stove  burner  is  hot 
given  that  hot  burners  are  orange  in  color. 

2.  Judging  the  consequences  inherent  in  a  possibly 
ambiguous  visual  scene.  Evaluating  a  route  to 
get  from  Boston  to  Miami  with  the  goal  being  to 
travel  the  fewest  miles. 

3.  Judging  the  consequences  in  an  ambiguous  visual 
scene  given  a  specific  standard.  Evaluating  an 
intercept  route  for  docking  a  spacecraft  with 
another  moving  craft,  when  the  goal  is  to  dock 
safely  using  the  least  amount  of  fuel. 

G.  EVALUATION  OF  AUDITORY  UNITS: 

1.  Judging  a  simple,  distinct,  auditory  stimulus  as 
being  similar  or  different  to  each  other  or  a 
specified  standard.  Evaluating  the  similarity 
of  two  tones  or  if  a  sound  is  similar  to  a 
target. 

2 .  Judging  a  moderately  complex  auditory  stimulus 
as  being  similar  or  different  to  each  other  or  a 
specified  standard.  Evaluating  whether  or  not  a 
dog's  bark  is  of  a  specified  breed  or  if  a  car 
horn  is  of  a  particular  model. 

3.  Judging  a  complex  auditory  stimuli  as  being 
similar  or  different  to  each  other  or  a 
specified  standard.  Evaluating  if  a  spoken  word 
matches  a  target  word  or  how  well  a  professional 
singer  reproduces  a  specified  song. 

H.  EVALUATION  OF  AUDITORY  CLASSES: 

I.  Evaluating  the  classification  of  a  non-complex 
auditory  stimulus  based  on  a  specified, 
well-defined,  attribute.  Determining  if  given 
tones  should  be  classed  as  "sounds  less  than 
5,000  Hz"  or  "sounds  between  10  and  10,000 
decibels" . 

2 .  Evaluating  the  classification  of  a  moderately 
complex  visual  item  based  on  a  specified 
attribute.  Determining  the  appropriateness  of 
classifying  a  dog's  bark  as  loud  or  soft. 

3.  Evaluating  the  classification  of  a  complex 
visual  item  based  on  a  specified  attribute. 
Determining  the  appropriateness  of  classifying  a 
particular  musical  piece  as  being  composed  by 
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Chopin  or  classifying  a  specific  submarine's 
sonar  return  as  an  "Alfa"  class  ship. 

I.  EVALUATION  OF  AUDITORY  RELATIONS: 

1.  Judging  given,  simple  relationships  among  a  few 
distinct  auditory  stimuli,  relative  to  a 
specified  criteria.  Given  two  notes,  evaluating 
them  in  terms  of  octave  differential  or  the 
relative  loudness  or  softness  of  each  of  the 
tones  in  decibels. 

2.  Judging  given,  more  involved  relationships  among 
a  few  auditory  stimuli,  relative  to  a  specified 
criteria.  Selecting  the  best  voice  among  several 
who  audition  based  on  range,  volume,  and  tonal 
quality. 

3.  Judging  given,  complex  relationships  among  a  few 
potentially  ambiguous  auditory  stimuli.  Given 
two  melodies,  select  the  one  which  is  "more 
pleasing." 

J.  EVALUATION  OF  AUDITORY  SYSTEMS: 

1.  Analyzing  the  integration  of  a  few  simple  sounds 
to  form  a  unique  whole.  Appraising  a  bugler's 
signal  to  charge  or  a  simple  drum  cadence. 

2.  Analyzing  the  integration  of  several  moderately 
complex  sounds.  Identifying  some  of  the 
principle  sounds  in  a  busy  bus  station  or 
interpreting  an  expected  message  on  the 
telephone. 

3.  Analyzing  the  arrangement  and  position  of 
various  stimuli  in  a  complex  auditory  system 
with  varying  degrees  of  background  clutter. 
Appraising  a  radio  communication  that  is 
partially  jammed  or  identifying  the  position  of 
a  submarine  from  several  auditory  inputs. 

K.  EVALUATION  OF  AUDITORY  TRANSFORMATIONS: 

1.  Appraising  the  impact  of  changes  on  a  simple 
auditory  stimulus  after  some  specific  changes 
occur.  Judging  a  tone  has  increased  in  pitch  or 
a  steady  tone  is  now  wavering. 

2 .  Appraising  the  result  of  several  changes  imposed 
on  a  moderately  complex  auditory  arrangement. 
Determining  the  impact  of  changing  the  key  of  a 
musical  selection  or  the  effect  of  increasing 
the  tempo  of  spoken  words  on  intelligibility. 

3.  Appraising  the  impact  of  subtle  changes  in  a 
complex  auditory  system.  Judging  verbal 
commands  made  by  a  pilot  under  high-g  stress  or 
interpreting  words  that  are  sung  rather  than 
spoken. 
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L.  EVALUATION  OF  AUDITORY  IMPLICATIONS: 

1.  Judging  suitability  of  the  consequences  inherent 
in  a  simple  auditory  stimulus  to  satisfy  a  given 
criteria.  Evaluating  the  adequacy  of  a  fire 
alarm  to  rouse  sleeping  tenants  or  the  adequacy 
of  a  warning  horn  to  alert  the  pilot  to  an 
inoperative  subsystem. 

2.  Judging  suitability  of  the  consequences  inherent 
in  a  moderately  complex  auditory  stimulus  to 
satisfy  a  given  criteria.  Evaluating  the  sound 
of  a  train  in  the  night  to  determine  if  it  is 
approaching  or  receding  or  determining  if  a 
dishwasher  is  operating  correctly  by  its  sound. 

3.  Judging  suitability  of  the  consequences  inherent 
in  a  complex  auditory  stimulus  to  satisfy  a 
given  criteria.  Evaluating  the  performance  of  a 
symphony  orchestra  to  determine  if  additional 
practice  is  required. 

M.  EVALUATION  OF  SYMBOLIC  UNITS: 

1.  Making  rapid  judgements  regarding  identification 
of  a  simple,  distinct,  symbol  relative  to  a 
specified  criteria.  Judging  that  two  given 
numbers  are  the  same  or  different  or  selecting 
all  words  in  a  list  containing  the  letter  "d." 

2.  Making  judgements  regarding  identification  of 
moderately  complex  symbols  relative  to  a 
specified  criteria.  Selecting  all  symbols  on  a 
map  of  cities  with  populations  greater  than 
50,000  people. 

3.  Making  judgements  regarding  identification  of 
complex  symbols  relative  to  a  specified 
criteria.  Selecting  key  words  from  a 
handwritten  note. 

N.  EVALUATION  OF  SYMBOLIC  CLASSES: 

1.  Judging  the  applicability  of  class  properties 
for  a  given  simple  symbol  based  on  a  specified 
criteria.  Given  a  number  (i.e.,  2)  and  several 
classes  with  associated  point  values  (i.e., 
primes  =  4  points,  even  *  2  points,  odd  =  1 
point) ,  classify  the  number  to  achieve  the  most 
points  (i.e.,  primes). 

2.  Judging  the  applicability  of  class  properties 
for  a  given  moderately  complex  symbol  based  on 
specified  criteria.  Given  a  specific  version 

of  a  new  product  logo,  determine  its  suitability 
for  projecting  the  desired  image  to  the  target 
market. 


3.  Judging  the  applicability  of  class  properties 
for  a  given  complex  symbol  based  on  specified 
criteria.  Given  a  person  with  a  specific  status 
symbol  (i.e.,  a  BMW),  determine  the  suitability 
of  that  person  as  typical  of  the  people  evoked 
by  the  symbol. 

O.  EVALUATION  OF  SYMBOLIC  RELATIONS: 

1.  Ascertaining  the  consistency  and  similarity 
among  simple  relationships  composed  of  a  few 
distinct  symbols.  Determining  if  the  conclusion 
of  specified  by  a  symbolic  equation  is  true  or 
false  (i.e.,  5-4=9). 

2.  Ascertaining  the  consistency  and  similarity  of 
more  involved  relationships  composed  of  several 
symbols.  Determining  if  a  given  passage  uses 
consistent  verb  tense. 

3.  Ascertaining  the  consistency  and  similarity  of 
complex  relationships  composed  of  many  symbols. 
Determining  the  truth  of  a  mathematical  proof  by 
verifying  all  the  supporting  calculations. 

P.  EVALUATION  OF  SYMBOLIC  SYSTEMS: 

1.  Estimating  the  suitability  of  aspects  of  a 
simple  symbolic  system  to  satisfy  a  given 
requirement.  Given  a  few  numbers  (i.e.,  102, 
945,  256,  220)  judge  which  number  is  most  unlike 
the  other  three  or  given  six  letters  (i.e.,  a, 
c,  e,  o,  u,  i)  specify  the  letter  that  does  not 
belong. 

2.  Estimating  the  suitability  of  aspects  of  a 
moderately  complex  symbolic  system  to  satisfy  a 
given  requirement.  Determining  the  quality 
(measured  as  resistance  to  being  broken  by  an 
enemy)  of  a  military  code. 

3.  Estimating  the  suitability  of  aspects  of  a 
complex  symbolic  system  to  satisfy  a  given 
requirement.  Determining  the  suitability  of  one 
language  over  another  for  communicating 
scientific  information. 

Q.  EVALUATION  OF  SYMBOLIC  TRANSFORMATIONS: 

1.  Judging  if  a  specific  ordering  of  simple, 
substitute  symbols  is  adequate  to  convey  the 
original  information.  Determining  if  specific 
words  can  be  made  by  rearranging  the  letters  of 
a  given  word. 

2.  Judging  if  a  specific  ordering  of  moderately 
complex,  substitute  symbols  is  adequate  to 
convey  the  original  information.  Determining 
which  of  several  label  designs  most  accurately 
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depicts  the  dangers  and  warnings  associated  with 
a  product. 

3.  Judging  if  a  specific  ordering  of  complex, 
substitute  symbols  is  adequate  to  convey  the 
original  information.  Determining  the  degree  of 
consistency  and  accuracy  in  translating  written 
material  from  Russian  to  English  (meaning  is  not 
involved) . 

R.  EVALUATION  OF  SYMBOLIC  IMPLICATIONS: 

1.  Evaluating  the  probability  or  inferences  drawn 
from  given  simple  symbolic  information.  Given  a 
simple  symbol,  select  which  of  three  objects  it 
best  represents  or  given  an  abbreviation  select 
the  word  it  best  represents  from  four 

alternatives. 

2.  Evaluating  the  probability  or  inferences  drawn 
from  given  moderately  complex  symbolic 
information.  Evaluating  how  far  your  car  can  go 
once  you  see  the  "FUEL  LOW"  warning  lamp  on  the 
dashboard  or  evaluating  the  outcome  of  selling 
or  holding  a  particular  stock  based  on  its 
climbing  price  over  several  days. 

3.  Evaluating  the  probability  or  inferences  drawn 
from  given  complex  symbolic  information.  Select 
the  best  symbol  to  represent  the  United  Nations 
from  several  given  candidates  or  based  on  the 
position  of  various  symbols  on  a  military 
tactical  battlefield  display,  determine  the 
probability  of  achieving  an  objective. 

S.  EVALUATION  OF  SEMANTIC  UNITS: 

1.  Judging  the  suitability  a  simple  idea  or  concept 
to  satisfy  a  given  requirement.  Evaluating  the 
meaning  of  non-complex  domain  elements  such  as 
"dog,"  "car,"  "bed,"  when  used  in  their  correct 
context . 

2.  Judging  the  suitability  of  a  moderately  complex 
idea  or  concept  to  satisfy  a  given  requirement. 
Evaluating  the  meaning  of  less-general  domain 
terms  such  as  "too  hot,"  "sell-short,"  or  other 
potentially  ambiguous  concepts  or  jargon  for  use 
in  a  given  context. 

3.  Judging  the  suitability  of  a  complex  idea  or 
concept  to  satisfy  a  given  requirement. 
Evaluating  meanings  of  abstract  terms  like 
"love,"  "beautiful,"  or  other  highly  subjective 
concepts  for  use  in  a  given  context. 

T.  EVALUATION  OF  SEMANTIC  CLASSES: 

1.  Judging  the  applicability  of  class  properties 
for  a  given  simple  idea  or  concept  based  on  a 
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specified  criteria.  Given  three  classes  (i.e., 
citrus,  fruit,  plant)  select  the  one  that  best 
applies  to  a  given  object  (i.e.,  an  orange). 

2.  Judging  the  applicability  of  class  properties 
for  a  given  moderately  complex  idea  or  concept 
based  on  a  specified  criteria.  Given  three  job 
titles  (i.e.,  trash  collector,  sanitation 
engineer,  or  city-street  custodian,  select  the 
one  that  best  applies  to  a  particular  job. 

3.  Judging  the  applicability  of  class  properties 
for  a  given  simple  idea  or  concept  based  on  a 
specified  criteria.  Given  three  properties 
(i.e.,  complex,  practical,  and  methodical) 
select  the  one  least  descriptive  of  a  particular 
concept  (i.e.,  philosophy). 

U.  EVALUATION  OF  SEMANTIC  RELATIONS: 

1.  Deciding  among  simple,  concrete  relationships 
among  a  few  concepts  or  situations  based  upon 
obvious  similarities  in  their  meanings. 
Identifying  simple  trends  in  data  (or  themes  in 
text)  or  completing  simple  analogies  (SIT  is  to 
CHAIR  as  LAY  is  to  BED) . 

2.  Ascertaining  less  certain  relationships  among  a 
few  concepts  based  upon  their  general 
similarity.  Evaluating  the  relationship  between 
violence  on  television  and  juvenile  delinquency 
or  the  relative  impact  of  education  on 
preventing  teen  pregnancy. 

3.  Ascertaining  complex  relationships  among 
potentially  ambiguous  concepts  or  ideas,  which 
may  also  have  dissimilar  meanings. 

U.  EVALUATION  OF  SEMANTIC  SYSTEMS: 

1.  Analyzing  the  internal  consistency  of  three  or 
more  integrated  unambiguous  concepts  which 
comprise  a  unique  whole.  Deciding  if  a  set  of 
rules  for  a  game  is  consistent  or  if  the 
procedures  for  handling  employee  grievances  are 
equitable  to  all. 

2.  Analyzing  the  internal  consistency  of  several 
moderately  complex  integrated  concepts  in 
potentially  ambiguous  contexts.  Judging  the 
reading-level  required  to  understand  a  newspaper 
article  or  the  consistency  of  a  contract  to 
purchase  a  medium-sized  weapon  system. 

3.  Analyzing  the  internal  consistency  of  many 
complex  integrated  concepts  in  generally 
ambiguous  contexts.  Appraising  the  consistency 
of  themes  in  a  literary  classic  or  the 
consistency  of  topics  in  a  politician's  speech. 
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V.  EVALUATION  OF  SEMANTIC  TRANSFORMATIONS: 

1.  Appraising  the  suitability  of  an  object  or  idea 
to  satisfy  specific  changes  imposed  on  a  simple 
conceptual  system  or  situation.  Given  four 
tools  (not  primarily  used  for  the  current  task) , 
concluding  which  tool  could  best  accomplish  the 
task  with  minimal  modification. 

2.  Appraising  the  suitability  of  an  object  or  idea 
to  satisfy  specific  changes  imposed  on  a 
moderately  complex  conceptual  system.  Concluding 
the  desirability  of  certain  design  changes  to 
meet  new  system  specifications  or  assessing  the 
viability  of  a  proposed  method  of  providing  mass 
transportation  for  a  growing  community. 

3.  Appraising  the  suitability  of  an  object  or  idea 
to  satisfy  subtle  or  major  changes  in  a  complex 
conceptual  system  or  situation.  Appraising  the 
organizational  changes  needed  to  implement  a  new 
long-range  strategy  or  deciding  the  outcomes  of 
adopting  a  new  advertising  strategy  for  a 
revised  product. 

X.  EVALUATION  OF  SEMANTIC  IMPLICATIONS: 

1  Judging  the  suitability  of  a  conclusion  based 
upon  complete  and  unambiguous  information. 
Evaluating  the  desirability  of  buying  product  A 
or  product  B  based  upon  a  complete  examination 
of  the  pros  and  cons  or  each  product. 

2.  Judging  the  suitability  of  a  conclusion  based 
upon  generally  complete  and  moderately  complex 
semantic  information.  Estimating  the 
probability  of  success  of  recommending  launch  of 
a  rocket  under  some  known  weather  conditions  and 
launch-window  characteristics . 

3.  Judging  the  suitability  of  a  conclusion  based 
upon  generally  incomplete  and  ambiguous  semantic 
information.  Deciding  national  fiscal  policy 
based  upon  leading  economic  indicators  or  the 
repercussions  of  unilateral  nuclear  disarmament. 

Y.  EVALUATION  OF  BEHAVIORAL  UNITS: 

1.  Judging  the  appropriateness  of  a  simple, 
distinct,  behavioral  action  in  a  non-complex 
situation.  Analyzing  the  appropriateness  of  a 
smile,  a  wink,  or  a  salute. 

2 .  Judging  the  appropriateness  of  a  moderately 
complex  behavioral  action  in  a  situation  with 
some  degree  of  ambiguity. 

3 .  Judging  the  appropriateness  of  a  complex 
behavioral  action  in  a  situation  with  some 
degree  of  ambiguity.  Identifying  correct 
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behaviors  or  responses  in  a  novel  situation, 
such  as  how  to  respond  upon  being  the  first  to 
arrive  at  a  massive  automobile  pile-up  with  many 
injuries . 

Z.  EVALUATION  OF  BEHAVIORAL  CLASSES: 

1.  Evaluating  the  classifications  of  behavioral 
items  by  a  few  simple  or  readily  apparent 
attributes.  Given  a  set  of  facial  expressions 
and  descriptions  of  each,  assess  the 
appropriateness  of  the  description  for  each 
expression. 

2.  Judging  the  classification  of  a  moderately 
complex  behavioral  activities  on  several 
potentially  ambiguous  or  not  readily  apparent 
dimensions . 

3.  Evaluating  the  classification  of  complex 
behavioral  activities  on  several  potentially 
ambiguous  or  not  readily  apparent  dimensions. 
Deciding  if  a  particular  version  of  choreography 
evokes  the  correct  emotion  or  if  group  of 
vociferous  demonstrators  can  be  classified  as  a 
"mob"  requiring  police  control. 

AA.  EVALUATION  OF  BEHAVIORAL  RELATIONS: 

1.  Ascertaining  simple  relationships  among  a  few 
distinct  behavioral  actions  or  situations. 
Evaluating  one's  basic  roles  in  a  family 
(mother,  wife,  and  breadwinner,  could  describe 
the  basic  roles  of  a  woman  in  a  family)  or  the 
relative  standing  of  two  politicians  on  a 
particular  issue. 

2 .  Ascertaining  moderately  complex  relationships 
among  potentially  ambiguous  behavioral  actions 
or  situations. 

3.  Ascertaining  complex  relationships  among 
potentially  ambiguous  behavioral  actions  or 
situations.  Appraising  an  organization's  status 
within  in  industry  (such  as  AT&T  in  the 
communications  industry)  and  the  power 
associated  with  that  status. 

BB.  EVALUATION  OF  BEHAVIORAL  SYSTEMS: 

1.  Analyzing  the  consistency  of  interactions  in  a 
simple  social  system.  Determining  if  certain 
actions  always  result  in  certain  outcomes. 

2.  Analyzing  the  consistency  of  interactions  in  a 
moderately  complex  social  system.  Assessing  the 
consistency  of  a  a  person's  words  and  their  body 
language  (i.e.,  does  the  mouth  say  one  thing 
while  the  body  says  the  opposite?) . 
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3.  Analyzing  the  interactions  and  acceptable 

actions  in  a  complex  social  system  with  varying 
degrees  of  ambiguity.  Appraising  the  consistency 
of  a  nation's  rhetoric  and  their  record  on  the 
issues. 

CC.  EVALUATION  OF  BEHAVIORAL  TRANSFORMATIONS: 

1.  Re-evaluating  a  simple  action  to  alter  its 
behavioral  value.  Deciding  the  appropriateness 
of  a  particular  action  (i.e.,  telling  an 
off-color  joke)  given  alternative  settings 
(i.e.,  a  locker  room  or  a  formal  dinner  party). 

2.  Re-evaluating  the  impact  of  changes  in  a 
moderately  complex  behavioral  activity. 
Determining  if  changing  a  particular  behavior 
(i.e.,  style  of  dress)  will  result  in  a  positive 
outcome  (i^e.,  gaining  more  respect). 

3.  Re-evaluating  the  impact  of  subtle  or  major 
changes  in  a  complex  behavioral  activity. 

Judging  how  an  ally  may  react  based  upon  a 
change  in  financial  aid  to  that  ally  or 
reconsidering  a  verdict  based  upon  new  evidence. 

DD.  EVALUATION  OF  BEHAVIORAL  IMPLICATIONS: 

1.  Judging  the  possible  consequences  suggested  in  a 
distinct  behavioral  action.  Judging  if  a  smile 
signifies  permission  to  approach  or  assessing 
the  audience's  interest  in  your  topic  by  the 
number  of  closed  eyes  and  open  mouths. 

2.  Appraising  the  probability  or  consistency  of 
possible  consequences  in  a  moderately  ambiguous 
behavioral  activity  or  situation.  Determining 
the  possible  outcomes  implied  by  the  "boss" 
inviting  you  to  dinner  (i.e.,  you  may  be 
receiving  a  raise  or  you  may  have  been  selected 
for  an  undesirable  assignment) . 

3.  Appraising  the  probability  or  consistency  of 
possible  consequences  in  an  ambiguous  behavioral 
activity  or  situation.  Deciding  the  chances  of 
getting  your  boss  to  give  you  a  raise  or  the 
probabilities  of  success  associated  with 
alternative  responses  to  an  adversary's 
aggressive  act. 


APPENDIX  B.  DEBUGGING  DOMAIN  DSAT  DATA 
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APPENDIX  B1 

DEBUGGING  DOMAIN  DSAT  DATA 
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0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

E 

A 

TRNS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

F 

L 

IMPL 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

OPER 

AVG  | 

|  8.0 

o 

• 

CO 

3.3 

0.0 

0.0 

o 

* 

o 

o 

0 

o 

0.0 

o 

• 

o 

o 

• 

o 

0.0 

0.0 

o 

• 

o 

o 

• 

o 

o 

• 

o 

G 

A 

UNTS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

H 

U 

CLSS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

I 

D 

RELS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

J 

I 

SYST 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

K 

T 

TRNS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

L 

Y 

IMPL 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

OPER 

AVG  | 

o 

. 

o 

o 

• 

o 

o 

• 

o 

o 

• 

o 

o 

. 

o 

o 

• 

o 

o 

• 

o 

o 

• 

o 

o 

• 

o 

0.0 

o 

• 

o 

o 

• 

o 

o 

• 

o 

0.0 

o 

• 

o 

M 

S 

UNTS 

10 

10 

3.3 

6.0 

8.0 

3.3 

6.0 

6.0 

3.3 

8.0 

8.0 

3.3 

8.0 

8.0 

3.3 

N 

Y 

CLSS 

8.0 

8.0 

6.7 

4.0 

4.0 

3.3 

2.0 

2.0 

3.3 

8.0 

8.0 

3.3 

6.0 

6.0 

3.3 

0 

M 

RELS 

6.0 

8.0 

6.7 

8.0 

8.0 

3.3 

6.0 

6.0 

3.3 

6.0 

8.0 

6.7 

10 

10 

3.3 

P 

B 

SYST 

8.0 

6.0 

10 

6.0 

6.0 

3.3 

6.0 

6.0 

3.3 

4.0 

4.0 

3.3 

4.0 

4.0 

3.3 

Q 

0 

TRNS 

4.0 

6.0 

3.3 

6.0 

6.0 

3.3 

2.0 

2.0 

3.3 

4.0 

4.0 

3.3 

6.0 

6.0 

3.3 

R 

L 

IMPL 

6.0 

6.0 

3.3 

6.0 

8.0 

3.3 

2.0 

4.0 

3.3 

2.0 

2.0 

3.3 

4.0 

4.0 

3.3 

OPER 

AVG  | 

|  7 . 0 

7.3 

so 

• 

tn 

o 

• 

VO 

6.7 

3.3 

a 

• 

o 

4.3 

3.3 

5.3 

5.7 

3.9 

6.3 

6.3 

u 

• 

S 

S 

UNTS 

6.0 

6.0 

3.3 

4.0 

6.0 

3.3 

2.0 

4.0 

3.3 

6.0 

6.0 

3.3 

4.0 

6.0 

3.3 

T 

E 

CLSS 

4.0 

4.0 

3.3 

4.0 

4.0 

3.3 

4.0 

4.0 

3.3 

4.0 

6.0 

3.3 

6.0 

4.0 

3.3 

U 

M 

RELS 

8.0 

8.0 

3.3 

6.0 

8.0 

3.3 

4.0 

2.0 

3.3 

2.0 

2.0 

3.3 

4.0 

4.0 

3.3 

V 

A 

SYST 

6.0 

6.0 

3.3 

8.0 

8.0 

3.3 

4.0 

4.0 

3.3 

6.0 

6.0 

3.3 

4.0 

4.0 

3.3 

W 

N 

TRAN 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

X 

T 

IMPL 

4.0 

4.0 

3.3 

6.0 

6.0 

3.3 

10 

10 

3.3 

6.0 

6.0 

3.3 

4.0 

4.0 

3.3 

OPER  AVG  | 5 . 6  5.6  3.3  5.6  6.4  3.3  4.8  4.8  3.3  4.8  5.2  3.3  4.4  4.4  3.3 

Y  B  UNTS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

Z  E  CLSS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

AA  H  RELS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

BB  A  SYST  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

CC  V  TRNS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

DD  L  IMPL  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

OPER  AVG  | 0 . 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 


140 


LEVEL  TWO  DATA  SUMMARY 

********************************************************************** 


CONTENT 


COGNITION 
F  C  D 


VISUAL 

AUDITORY 

SYMBOLIC 

SEMANTIC 

BEHAVORAL 


8.0  8.0  3.3 
0.0  0.0  0.0 
7.0  7.3  5.6 
5.6  5.6  3.3 
0.0  0.0  0.0 


OPER  AVG  | 6 . 9  7.0  4.1 


MEMORY 
F  C  D 


0.0  0.0  0.0 
0.0  0.0  0.0 
6.0  6.7  3.3 
5.6  6.4  3.3 
0.0  0.0  0.0 


5.8  6.5  3.3 


DIVT  PRODN 
F  C  D 


0.0  0.0  0.0 
0.0  0.0  0.0 
4.0  4.3  3.3 

4.8  4.8  3.3 

0.0  0.0  0.0 


4.4  4.6  3.3 


CONV  PRODN 
F  C  D 


0.0  0.0  0.0 
0.0  0.0  0.0 
5.3  5.7  3.9 

4.8  5.2  3.3 

0.0  0.0  0.0 


5.1  5.4  3.6 


EVALUATION 
F  C  D 


0.0  0.0  0.0 
0.0  0.0  0.0 

6.3  6.3  3.3 

4.4  4.4  3.3 

0.0  0.0  0.0 


5.4  5.4  3.3 


LEVEL  THREE  DATA  SUMMARY 

********************************************************* 


OPERATOR 

1  F 

C 

D 

N 

|  FINAL  RATING 

COGNITION 

6.9 

7.0 

4.1 

13 

5.4 

MEMORY 

5.8 

6.5 

3.3 

11 

4.7 

DIVT.  PROD. 

4.4 

4.6 

3.3 

11 

3.9 

CONV.  PROD. 

5.1 

5.4 

3.6 

11 

4.4 

EVALUATION 

5.4 

5.4 

3.3 

11 

4.3 

DOMAIN  AVERAGE  1 

|  5.5 

CO 

• 

in 

3.5 

57  1 

1  4.4 

io  3  -  r+  w  t  a-ituaaw^c/) 
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APPENDIX  B2 

DEBUGGING  DOMAIN  DSAT  LEVEL  1,  2,  AND  3  GRAPHS 


DSAT  Level  3  Data  Summary 
for  Debugging  Domain 

Doamin  Suitability  Index  Score  ■  4.4 

10 

9 
8 
7 
6 
5 
4 
3 
2 
1 
0 

CognltionMemory  Dv.  Prd.  Cv.  Prd. Evaluation 

Domain  Operators 

57  Domain  Elements  Identified 


Dimension 


n — n 
n  ii  ii 

II  II  H 

ii  ii  ii 

1  H  II 


II  I 
II  I 
II  I 


Combined  Rating 
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DSAT  Level  2  Data  Summary 
for  Debugging  Domain 


Cognition  Memory  Dv.  Prd.  Cv.  Prd.  Evaluation 


Domain  Operators 

57  Domain  Elements  Identified 

DSAT  Level  1  -  Cognition  Operator 
for  Debugging  Domain 


Domain  Contents 

13  Content-Products  Identified 


co)  (/)+^  (S  c"D  m  ‘-'o  qc  ro  +-  —  c  o> 
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DSAT  Level  1  -  Memory  Operator 
for  Debugging  Domain 


s 

t 

a 

n 

d 

a 

r 

d 

R 

a 

t 

i 


Visual  Auditory  Symbolic  Semantic  Behavioral 


Domain  Contents 


Dimension 


Frequency 


ISvSM  Criticality 


l _ j  Difficulty 


11  Content-Products  Identified 


DSAT  Level  1  -  Divergent  Production 
for  Debugging  Domain 


{ 

— 

■n 

- 

- 

— 

H 

j - 

1 

J 

i 

_ 1 _ 

— 

i 

Visual  Auditory  Symbolic  Semantic  Behavioral 


Domain  Contents 


Dimension 


Frequency 


Criticality 


1  . 1  Difficulty 


11  Content-Products  Identified 


<QD  —  «-^0)3D  ai  W  Oi3  ft)  '♦W  (03  —  W  30  Q.~|  ft)  £13  ft) 
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DSAT  Level  1  -  Convergent  Production 
for  Debugging  Domain 


Visual  Auditory  Symbolic  Semantic  Behavioral 


Domain  Contents 

11  Content-Products  Identified 


DSAT  Level  1  -  Evaluation  Operator 
for  Debugging  Domain 


1 

_ 

1 

II 

B 

i 

| 

i 

i 

m 

i 

mm 

|i 

n 

l 

i 

_ 

li 

■  ■ 

Visual  Auditory  Symbolic  Semantic  Behavioral 


Domain  Contents 


Dimension 


Frequency 


ISMSI  Criticality 
i...  .1  Difficulty 


11  Content-Products  Identified 


APPENDIX  C.  RESEARCH  PROPOSAL  DOMAIN  DSAT  DATA 
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APPENDIX  Cl 


RESEARCH  PROPOSAL  SUBMISSION  DOMAIN  DSAT  DATA 
LEVEL  ZERO  DATA  SUMMARY 

********************************************************************* 
QUESTION/ I  COGNITION  MEMORY  DIVT  PRODN  CONV  PRODN  EVALUATION 


PROD- 

■cont| 

F 

C 

D 

F 

C 

D 

F 

c 

D 

F 

c 

D 

F 

C 

D 

A 

V 

UNTS 

10 

10 

3 . 3 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

2.0 

2.0 

10 

0.0 

0.0 

0.0 

B 

I 

CLSS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

C 

S 

RELS 

0.0 

0 . 0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

4.0 

4.0 

3.3 

0.0 

0.0 

0.0 

D 

u 

SYST 

0.0 

0.0 

0.0 

6.0 

2.0 

6.7 

0 . 0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

E 

A 

TRNS 

0.0 

0 . 0 

0 . 0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

F 

L 

IMPL 

2.0 

4.0 

3.3 

0.0 

0.0 

0.0 

0. 0 

0.0 

0.0 

2 . 0 

2.0 

3.3 

4.0 

4.0 

6.7 

OPER  AVG | 6 . 0 

7 . 0 

3 . 3 

6.0 

2 . 0 

6.7 

0.0 

0.0 

0.0 

2.7 

2.7 

5.4 

4 . 0 

4.0 

6.7 

G 

A 

UNTS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

H 

u 

CLSS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

I 

D 

RELS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

J 

I 

SYST 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

K 

T 

TRNS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

L 

Y 

IMPL 

4.0 

8.0 

10 

4.0 

4.0 

10 

0.0 

0.0 

0.0 

4.0 

4.0 

10 

4.0 

4.0 

3.3 

OPER  AVG 

|  0 . 7 

1.3 

1.7 

0.7 

0.7 

1.7  0.0 

0.0 

0.0 

0.7 

0.7  1.7 

0.7  0.7 

0.6 

M 

S 

UNTS 

4.0 

4 . 0 

3 . 3 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

N 

Y 

CLSS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0 

M 

RELS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

4.0 

4.0 

6.7 

P 

B 

SYST 

8.0 

8.0 

10 

6.0 

4.0 

6.7 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Q 

0 

TRNS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

R 

L 

IMPL 

6.0 

6.0 

10 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

4.0 

8.0 

10 

0.0 

0.0 

0.0 

OPER  AVG 

1  6.7 

6.7 

7.8 

6.0 

4.0 

6.7 

0.0 

0.0 

0.0 

4.0 

8.0 

10 

4.0 

4.0 

6.7 

S 

S 

UNTS 

10 

10 

10 

10 

10 

10 

6 . 0 

8.0 

10 

4.0 

4.0 

10 

6.0 

8.0 

10 

T 

E 

CLSS 

6.0 

8.0 

10 

10 

10 

10 

4 . 0 

8.0 

10 

2.0 

4.0 

6.7 

6.0 

8.0 

10 

U 

M 

RELS 

6.0 

10 

10 

10 

10 

10 

8.0 

10 

10 

6.0 

10 

10 

6.0 

10 

10 

V 

A 

SYST 

6.0 

10 

10 

6.0 

10 

10 

10 

10 

10 

6.0 

8.0 

10 

2.0 

4.0 

10 

W 

N 

TRNS 

6.0 

10 

10 

6.0 

10 

10 

6.0 

10 

10 

8.0 

8.0 

10 

6.0 

6.0 

10 

X 

T 

IMPL 

4.0 

10 

10 

8.0 

10 

10 

8 

10 

10 

4.0 

6.0 

10 

6.0 

8.0 

10 

OPER  AVG 

|  6.3 

9.7 

10 

r> 

• 

00 

10 

10 

7.0 

9.3 

10 

5.0 

6.7 

9.5 

5.3 

7.3 

10 

Y  B  UNTS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
Z  E  CLSS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
AA  H  RELS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
BB  A  SYST  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
CC  V  TRNS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
DD  L  IMPL  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

OPER  AVG | 0 . 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
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LEVEL  TWO  DATA  SUMMARY 


CONTENT 


COGNITION  MEMORY  DIVT  PRODN  CONV  PRODN  EVALUATION 

FCDFCDFCDFCDFCD 


VISUAL 

AUDITORY 

SYMBOLIC 

SEMANTIC 

BEHAVRL 


6.0  7.0  3.3 
0.7  1.3  1.7 
6.7  6.7  7.8 

6.3  9.7  10 

0.0  0.0  0.0 


6.0  2.0  6.7 
0.7  0.7  1.7 
6.0  4.0  6.7 

8.3  10  10 

0.0  0.0  0.0 


0.0  0.0  0.0 
0.0  0.0  0.0 
0.0  0.0  0.0 
7.09.3  10 

0.0  0.0  0.0 


2.7  2.7  5.4 
0.7  0.7  1.7 
4.08.0  10 
5.0  6.7  9.5 
0.0  0.0  0.0 


4.0  4.0  6.7 
0.7  0.7  0.6 
4.0  4.0  6.7 

5.37.3  10 

0.0  0.0  0.0 


OPER  AVG  5.1  6.2  5.7  5.3  4.2  6.3  7.0  9.3  10  3.1  4.5  6.6  3.5  4.0  6.0 


LEVEL  THREE  DATA  SUMMARY 

******************************************************** 


OPERATOR 

1  F 

C 

D 

N 

|  FINAL  RATING 

COGNITION 

5.1 

6.2 

5.7 

12 

5.7 

MEMORY 

5.3 

4.2 

6.3 

9 

5.3 

DIVT.  PROD. 

7.0 

9.3 

10 

6 

9.1 

CONV.  PROD. 

3.1 

4.5 

6.6 

11 

5.4 

EVALUATION 

3.5 

4.0 

6.0 

9 

4.9 

DOMAIN  AVERAGE 

1  4.8 

5.6 

6.9 

46  1 

1  7.2 

C/)-*-ajc'0n3i-'0  Qlns*-  —  c  cn 
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APPENDIX  C2 

RESEARCH  PROPOSAL  SUBMISSION  DOMAIN  DSAT  LEVEL  1,  2,  AND  3 


DSAT  Level  3  Data  Summary 
for  Research  Proposal  Domain 


Domain  Operators 

46  Domain  Elements  Identified 


GRAPHS 


DSAT  Level  2  Data  Summary 
for  Research  Proposal  Domain 


Cognition  Memory  Dv.  Prd.  Cv.  Prd.  Evaluation 


Domain  Operators 

46  Domain  Elements  Identified 


DSAT  Level  1  -  Cognition  Operator 
for  Research  Proposal  Domain 


Visual  Auditory  Symbolic  Semantic  Behavioral 


Domain  Contents 

12  Content-Products  Identified 


(A  d  --►6)  I]  Q.-idianw^w 
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DSAT  Level  1  -  Memory  Operator 
for  Research  Proposal  Domain 


Domain  Contents 


Dimension 
Hi  Frequency 
EH  Criticality 
1  I  Difficulty 


9  Content-Products  Identified 


DSAT  Level  1  -  Divergent  Production 
for  Research  Proposal  Domain 


_ _ 

1 

1 

1 

1 

- 

I 

. 

- 

■ 

. _ 1 - 

_ 1 _ 

( 

_ 1 _ _ 

Visual  Auditory  Symbolic  Semantic  Behavioral 


Domain  Contents 


Dimension 


Frequency 


EM  Criticality 
! _ J  Difficulty 


6  Content-Products  Identified 


DSAT  Level  1  -  Convergent  Production 
for  Research  Proposal  Domain 
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r  4 
-—-a 

---‘-41 


Dimension 


Frequency 


Criticality 
I _ ]  Difficulty 


I  Auditory  Symbolic  Semantic  Behavioral 

Domain  Contents 

11  Content-Products  Identified 


DSAT  Level  1  -  Evaluation  Operator 
for  Research  Proposal  Domain 


Dimension 


■1  Frequency 
PMl  Criticality 
I  I  Difficulty 


Visual  Auditory  Symbolic  Semantic  Behavioral 

Domain  Contents 

9  Content-Products  Identified 


APPENDIX  D.  GIFTED  CHILD  ASSESSMENT  DOMAIN  DSAT  DATA 
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APPENDIX  D1 

GIFTED  CHILD  ASSESSMENT  DOMAIN  DSAT  DATA 
LEVEL  ZERO  DATA  SUMMARY 

********************************************************************** 
QUESTION/ I  COGNITION  MEMORY  DIVT  PRODN  CONV  PRODN  EVALUATION 


PROD- 

-CONT  j 

F 

C 

D 

F 

C 

D 

F 

C 

D 

F 

c 

D 

F 

c 

D 

A 

V 

UNTS 

6.0 

8.0 

10 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

6.0 

10 

10 

B 

I 

CLSS 

4 . 0 

8.0 

6.7 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

6.0 

10 

3.3 

C 

s 

RELS 

6.0 

6.0 

6.7 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

4.0 

10 

3.3 

D 

u 

SYST 

2.0 

2.0 

3 . 3 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

4.0 

8 . 0 

3 . 3 

E 

A 

TRNS 

2 . 0 

4.0 

3.3 

0.0 

0. 0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

F 

L 

IMPL 

4 . 0 

8.0 

6.7 

0.0 

0 . 0 

0.0 

0 . 0 

0.0 

0.0 

0.0 

0.0 

0.0 

4.0 

8.0 

3.3 

OPER  AVG| 

4.0 

6.0 

6.1 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

4.8 

9.2 

4.6 

G 

A 

UNTS 

8.0 

8.0 

6.7 

0.0 

0.0 

0.0 

2.0 

6.0 

3.3 

2.0 

4.0 

3.3 

8.0 

8.0 

6.7 

H 

U 

CLSS 

6.0 

8.0 

6.7 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

2.0 

4.0 

3.3 

2 . 0 

4.0 

3.3 

I 

D 

RELS 

6.0 

8 . 0 

6.7 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

2 . 0 

4.0 

3.3 

J 

I 

SYST 

4.0 

6.0 

6.7 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

2.0 

4.0 

3.3 

0.0 

0.0 

0.0 

K 

T 

TRNS 

6.0 

6.0 

6.7 

0.0 

0.0 

0.0 

2.0 

4.0 

3.3 

2.0 

4.0 

3.3 

0.0 

0.0 

0.0 

L 

Y 

IMPL 

8.0 

8.0 

10 

0.0 

0.0 

0.0 

2.0 

4.0 

3.3 

2.0 

4.0 

3.3 

6.0 

8.0 

3.3 

OPER  AVG | 

6.3 

7.3 

7.3 

0.0 

0.0 

0.0 

2.0 

4.7 

3.3 

2.0 

4.0 

3.3 

4.5 

6.0 

4.2 

M 

S 

UNTS 

6.0 

8.0 

10 

10 

10 

3.3 

6.0 

6.0 

3.3 

0.0 

0.0 

0.0 

6.0 

8.0 

3.3 

N 

Y 

CLSS 

8.0 

8.0 

10 

0.0 

0.0 

0.0 

4.0 

4.0 

3.3 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0 

M 

RELS 

8.0 

10 

10 

10 

10 

6.7 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

6.0 

10 

3.3 

P 

B 

SYST 

4.0 

6.0 

6.7 

10 

10 

10 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

6.0 

8.0 

3.3 

Q 

0 

TRNS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

6.0 

8.0 

6.7 

R 

L 

IMPL 

6.0 

8.0 

6.7 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

OPER  AVG 

|  6.4 

8.0 

8.7 

10 

10 

6.7 

5.0 

5.0 

3.3 

0.0 

0.0 

0.0 

6.0 

8.5 

4.2 

S 

S 

UNTS 

6  0 

10 

10 

6.0 

6.0 

6.7 

6.0 

6.0 

6.7 

8.0 

10 

6.7 

8.0 

10 

6.7 

T 

E 

CLSS 

4.0 

8.0 

10 

6 .  C 

6.0 

6.7 

0.0 

0.0 

0.0 

8.0 

10 

6.7 

8.0 

10 

6.7 

U 

M 

RELS 

2 . 0 

8.0 

10 

10 

8.0 

3.3 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

6.0 

6.0 

3.3 

V 

A 

SYST 

2.0 

6.0 

6.7 

8.0 

8.0 

3.3 

0.0 

0.0 

0.0 

8.0 

10 

6.7 

4.0 

4.0 

3.3 

W 

N 

TRNS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

X 

T 

IMPL 

2 . 0 

4.0 

6.7 

10 

8.0 

3.3 

4 . 0 

6.0 

6.7 

6.0 

10 

10 

4.0 

6.0 

3.3 

OPER  AVG | 3 . 2 

7.2 

8.7 

8.0 

7.2 

4.7 

5.0 

6.0 

6.7 

7.5 

10 

7.5 

6.0 

7.2 

4.7 

Y 

B 

UNTS 

8.0 

8 . 0 

6.7 

4.0 

4.0 

3.3 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

4.0 

8.0 

6.7 

Z 

E 

CLSS 

8.0 

8.0 

6.7 

4.0 

4.0 

3.3 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

AA 

H 

RELS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

BB 

A 

SYST 

8 . 0 

8.0 

6.7 

2 . 0 

2.0 

3.3 

2 . 0 

2 . 0 

3.3 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

CC 

V 

TRNS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

DD 

L 

IMPL 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

OPER  AVG 

|  8 . 0 

8.0 

6.7 

3 . 3 

3 . 3 

3.3 

2 . 0 

2 . 0 

3.3 

0.0 

0.0 

0.0 

4.0 

8.0 

6.7 

LEVEL  TWO  DATA  SUMMARY 


A*********************************************************** 


PRODUCT 

COGNITION 

F  C  D 

MEMORY 

F  C  D 

DIVT  PRODN 
F  C  D 

CONV  PRODN 
F  C  D 

EVALUATION 

F  C  D 

VISUAL 

4 . 0 

6.0 

6.1 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

4.8 

9.2 

4.6 

AUDITORY 

6.3 

7.3 

7.3 

0 . 0 

0.0 

0.0 

2.0 

4.7 

3.3 

2.0 

4.0 

3.3 

4 . 5 

6.0 

4.2 

SYMBOLIC 

6.4 

8.0 

8.7 

10 

10 

6 . 7 

5.0 

5.0 

3.3 

0.0 

0.0 

0.0 

6.0 

8.5 

4.2 

SEMANTIC 

3.2 

7.2 

8.7 

8 . 0 

7.2 

4 . 7 

5.0 

6.0 

6.7 

7.5 

10 

7.5 

6.0 

7.2 

4.7 

BEHAVRL 

8.0 

8.0 

6.7 

3 . 3 

3.3 

3 . 3 

2.0 

2 . 0 

3 . 3 

0.0 

0.0 

0.0 

4.0 

8.0 

6.7 

OPER  AVG 

1  5.6 

7 . 3 

7.5 

'O 

• 

6 . 8 

4 . 9 

3 . 5 

4.4 

4.2 

2.4 

3.5 

2.7 

5.1 

7.8 

4.9 

LEVEL  THREE  DATA  SUMMARY 


OPERATOR  [  F  C  D  N  |  FINAL  RATING 


COGNITION  5.6  7.3  7.5  25  7.3 

MEMORY  7.1  6.8  4.9  11  5.7 

DIVT.  PROD.  3.5  4.4  4.2  8  4.1 

CONV.  PROD.  2.4  3.5  2.7  9  3.0 

EVALUATION  5.1  7.8  4.9  19  6.0 

DOMAIN  AVERAGE  |4.7  6.0  4.8  72  |  5.2 


c  o> 
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APPENDIX  D2 

GIFTED  CHILD  ASSESSMENT  DOMAIN  DSAT  LEVEL  1,  2,  and  3  GRAPHS 


DSAT  Level  3  Data  Summary 
for  Gifted  Child  Assessment  Domain 

Domain  Suitability  Index  Score  ■  5.2 


II  II 


U  II 


II 


Dimension 


Combined  Rating 


CognltlonMamory  Dv.  Prd.  Cv.  Prd. Evaluation 

Domain  Operators 

72  Domain  Elements  Identified 


DSAT  Level  2  Data  Summary 
for  Gifted  Child  Assessment  Domain 


10- 


Cognition  Memory  Ov.  Prd.  Cv.  Prd.  Evaluation 


Domain  Operators 

72  Domain  Elements  Identified 


DSAT  Level  1  -  Cognition  Operator 
for  Gifted  Child  Assessment  Domain 


10 


Visual  Auditory  Symbolic  Semantic  Behavioral 


Domain  Contents 


Dimension 


Frequency 


SSM  Criticality 


[_.J  Difficulty 


25  Content-Products  Identified 


DSAT  Level  1  -  Memory  Operator 
for  Gifted  Child  Assessment  Domain 


s 

t 

a 

n 

d 

a 

r 

d 

R 

a 

t 

i 

n 

g 


Visual  Auditory  Symbolic  Semantic  Behavioral 


Domain  Contents 


Dimension 


HI  Frequency 
M  Criticality 
1  Difficulty 


11  Content-Products  Identified 


DSAT  Level  1  -  Divergent  Production 
for  Gifted  Child  Assessment  Domain 


S  10 


Visual  Auditory  Symbolic  Semantic  Behavioral 


Domain  Contents 

8  Content-Products  Identified 


flj  C  "O  CtJ  »-  TJ  OC  CO  **  —  CD)  €/)+-CflCTJCd»-TJ  DC  fO  CO) 
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DSAT  Level  1  -  Convergent  Production 
for  Gifted  Child  Assessment  Domain 


Visual  Auditory  Symbolic  Semantic  Behavioral 


Domain  Contents 

9  Content-Products  Identified 


DSAT  Level  1  -  Evaluation  Operator 
for  Gifted  Child  Assessment  Domain 


Visual  Auditory  Symbolic  Semantic  Behavioral 


Domain  Contents 


Dimension 


Frequency 


STj  Criticality 


Difficulty 


19  Content-Products  Identified 


appendix  e.  subject  informed  consent  form 
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APPENDIX  E 

SUBJECT  INFORMED  CONSENT  FORM 

April  3,  1989 


To  Whom  It  May  Concern: 

The  results  of  the  research  project  you  are  participating  in  today  will 
provide  information  to  determine  the  reliability  and  aid  in  the 
refinement  of  a  domain  suitability  analysis  tool.  During  one 
experimental  session  of  approximately  one  hour  and  forty-five  minutes 
you  will  be  asked  to  complete  a  questionnaire  to  determine  and  rate 
various  aspects  of  information  you  use  and  the  mental  processes  you  use 
when  performing  the  task  of  debugging  a  small  (less  than  200  lines  of 
code)  non-complex  computer  program  with  specific  inputs  and  outputs. 
Potential  risks  associated  with  this  experiment  are  similar  to  any 
rating  task.  You  will  be  assigned  a  random  identification  number  and 
this  number  will  be  used  to  identify  your  data;  your  name  will  not 
appear  in  any  records  of  this  experiment.  If  you  have  any  questions, 
please  feel  to  contact  the  experimenter,  Jay  Horn,  at  233-0307  or  Dr. 
Richard  Koubek  at  873-2701.  At  the  conclusion  of  this  research,  a 
summary  of  group  results  will  be  avilable  from  the  experimenter  upon 
request. 


I  have  read  the  above  information  and  understand  that  participation  is 
voluntary,  refusal  to  participate  will  involve  no  penalty  or  loss  of 
benefits  to  which  I  am  otherwise  entitled  and  I  may  discontinue 
participation  at  any  time  without  penalty  or  loss  of  benefits  to 
which  I  am  otherwise  entitled.  I  realize  completion  of  the  tasks 
implies  my  consent  to  participate. 


Signature  Date 


APPENDIX  F 
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APPENDIX  FI 

LEVEL  ZERO  DATA  SUMMARY 

SUBJECT  1  9  YEARS  EXPERIENCE 

*  ★  *r  x  ★***★**★★*******★★★***★★*•**■*★**★★★★★★★*★**★★**★★★**★*★★**★*★★★*★★★* 

QUESTION/  COGNITION  MEMORY  DIVT  PRCDN  CONV  PRODN  EVALUATION 

PROD-CONT  FCDFCDFCDFCDFCD 


A  V  UNTS  10  10  3.3  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
B  I  CLSS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
C  S  RELS  6.0  6.0  3.3  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
D  U  SYST  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
E  A  TRNS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
F  L  IMPL  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 


OPER  AVG  |8.0  8.0  3.3  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 


G  A  UNTS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
H  U  CLSS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
I  D  RELS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
J  I  SYST  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
K  T  TRNS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  O.G  0.0  0.0  0.0  0.0  0.0  0.0 
L  Y  IMPL  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 


OPER  AVG  | 0 . 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

M  S  UNTS  10  10  3.3  6.0  8.0  3.3  6.0  6.0  3.3  8.0  8.0  3.3  8.0  8.0  3.3 
N  Y  CLSS  8.0  8.0  6.7  4.0  4.0  3.3  2.0  2.0  3.3  8.0  8.0  3.3  6.0  6.0  3.3 
O  M  RELS  6.0  8.0  6.7  8.0  8.0  3.3  6.0  6.0  3.3  6.0  8.0  6.7  10  10  3.3 
P  B  SYST  8.0  6.0  10  6.0  6.0  3.3  6.0  6.0  3.3  4.0  4.0  3.3  4.0  4.0  3.3 
Q  O  TRNS  4.0  6.0  3.3  6.0  6.0  3.3  2.0  2.0  3.3  4.0  4.0  3.3  6.0  6.0  3.3 
R  L  IMPL  6.0  6.0  3.3  6.0  8.0  3.3  2.0  4.0  3.3  2.0  2.0  3.3  4.0  4.0  3.3 


OPER  AVG  | 5 . 6  5.6  3.3  5.6  6.4  3.3  4.8  4.8  3.3  4.8  5.2  3.3  4.4  4.4  3.3 

Y  B  UNTS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

Z  E  CLSS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

AA  H  RELS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

BB  A  SYST  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

CC  V  TRNS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

DD  L  IMPL  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

OPER  AVG  | 0 . 0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
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APPENDIX  F2 

LEVEL  ZERO  DATA  SUMMARY 

SUBJECT  2  8  YEARS  EXPERIENCE 

********  ************************************************************** 
QUESTION/ I  COGNITION  MEMORY  DIVT  PRODN  CONV  PRODN  EVALUATION 


PROD- 

-CONT  | 

F 

c 

D 

F 

c 

D 

F 

c 

D 

F 

C 

D 

F 

C 

D 

A 

V 

UNTS 

10 

10 

3.3 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

B 

I 

CLSS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 
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OPER  AVG | 6 . 8  8.0  6.7  6.8  6.4  5.3  0.0  0.0  0.0  7.0  7.0  4.2  7.7  7.3  3.9 

S  S  UNTS  6.0  8.0  3.3  6.0  6.0  3.3  0.0  0.0  0.0  6.0  4.0  6.7  6.0  8.0  3.3 
T  E  CLSS  2.0  4.0  3.3  6.0  4.0  3.3  0.0  0.0  0.0  4.0  4.0  3.3  4.0  4.0  3.3 
U  M  RELS  4.0  4.0  6.7  8.0  6.0  6.7  0.0  0.0  0.0  0.0  0.0  0.0  6.0  6.0  3.3 

V  A  SYST  4.0  6.0  3.3  8.0  6.0  3.3  0.0  0.0  0.0  2.0  2.0  6.7  4.0  4.0  3.3 
W  N  TRAN  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
X  T  IMPL  4.0  6.0  6.7  4.0  6.0  6.7  10  10  6.7  4.0  4.0  3.3  6.0  2.0  6.7 
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Y  B  UNTS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
Z  E  CLSS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
AA  H  RELS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
BB  A  SYST  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
CC  V  TRNS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  n .  0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
DD  L  IMPL  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

OPER  AVG |0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
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• 

oo 

5.3 

6.0 

6.8 

5.3 

Y 

B 

UNTS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Z 

E 

CLSS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

AA 

H 

RELS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

BB 

A 

SYST 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

CC 

V 

TRNS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

DD 

L 

IMPL 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

OPER  AVG 

1  0. 0 

0.0 

0 . 0 

0.0 

0.0 

0.0 

0 . 0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 
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APPENDIX  F5 

LEVEL  ZERO  DATA  SUMMARY 

SUBJECT  5  5  YEARS  EXPERIENCE 

********************************************************************** 
QUESTION/)  COGNITION  MEMORY  DIVT  PRODN  CONV  PRODN  EVALUATION 


PROD- 

-CONT  | 

F 

c 

D 

F 

c 

D 

F 

C 

D 

F 

C 

D 

F 

c 

D 

A  V 

UNTS 

10 

10 

6.7 

2 . 0 

4.0 

10 

0.0 

0.0 

0.0 

6.0 

8 . 0 

6.7 

4.0 

2.0 

6.7 

B  I 

CLSS 

6.0 

4 . 0 

6.7 

6.0 

8 . 0 

6.7 

2 . 0 

8 . 0 

3.3 

0.0 

0.0 

0.0 

2.0 

2.0 

3.3 

C  S 

RELS 

0.0 

0.0 

0.0 

6.0 

4.0 

6.7 

0.0 

0.0 

0.0 

2.0 

2.0 

3.3 

4.0 

6.0 

6.7 

D  U 

SYST 

0.0 

0.0 

0.0 

0.0 

0.0 

0 . 0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

E  A 

TRNS 

8 . 0 

10 

10 

2 . 0 

4.0 

10 

0.0 

0 . 0 

0.0 

0.0 

0.0 

0.0 

2 . 0 

6.0 

3.3 

F  L 

IMPL 

4.0 

4 . 0 

6.7 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

4.0 

8.0 

10 

OPER  AVG | 

7.0 

7.0 

7.5 

4.0 

5.0 

8 . 4 

2 . 0 

8 . 0 

3.3 

4.0 

5.0 

5.0 

3.2 

4.8 

6.0 

G  A 

UNTS 

0.0 

0.0 

0.0 

0.0 

0 . 0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

H  U 

CLSS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

I  D 

RELS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

J  I 

SYST 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

K  T 

TRNS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

L  Y 

IMPL 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

OPER  AVG | 0 . 0 

0.0 

0.0 

0.0 

0.0 

0.0 

o 

• 

o 

0.0 

0.0 

0.0 

o 

• 

o 

0.0 

0.0 

0.0 

0.0 

M  S 

UNTS 

10 

8.0 

3.3 

10 

10 

3.3 

4.0 

8.0 

6.7 

6.0 

8.0 

6.7 

4.0 

8.0 

10 

N  Y 

CLSS 

6.0 

4.0 

6.7 

4.0 

6.0 

6.7 

6.0 

6.0 

6.7 

2.0 

6.0 

6.7 

6.0 

10 

3.3 

0  M 

RELS 

6.0 

10 

10 

2.0 

6.0 

10 

2.0 

6.0 

10 

6.0 

10 

10 

8.0 

10 

10 

P  B 

SYST 

2,0 

4.0 

6.7 

8.0 

8.0 

6.7 

0.0 

0.0 

0.0 

4.0 

6.0 

10 

8.0 

6.0 

3.3 

Q  0 

TRNS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

R  L 

IMPL 

6.0 

8.0 

6.7 

10 

8.0 

3.3 

4.0 

2.0 

6.7 

2.0 

6.0 

10 

4.0 

4.0 

3.3 

OPER  AVG |  6 . 0 

6.8 

6.7 

6.8 

7.6 

6.0 

4.0 

5.5 

7.5 

4.0 

7.2 

8.7 

6.0 

7.6 

6.0 

S  S 

UNTS 

4.0 

6.0 

6.7 

2.0 

2.0 

6.7 

4.0 

6.0 

10 

2.0 

4.0 

3.3 

4.0 

4.0 

6.7 

T  E 

CLSS 

8.0 

8.0 

3.3 

4.0 

6.0 

3 . 3 

4.0 

2.0 

3.3 

4.0 

4.0 

6.7 

4.0 

6.0 

3.3 

U  M 

RELS 

2 . 0 

2.0 

6.7 

6.0 

4.0 

3.3 

6.0 

2.0 

3.3 

4.0 

6.0 

3.3 

6.0 

6.0 

6.7 

V  A 

SYST 

8.0 

10 

6.7 

8.0 

10 

3 . 3 

4.0 

6.0 

6.7 

8.0 

10 

3.3 

0.0 

0.0 

0.0 

W  N 

TRAN 

0.0 

0.0 

0.0 

4.0 

6.0 

6.7 

4.0 

8.0 

6.7 

4.0 

4.0 

3.3 

2.0 

4.0 

3.3 

X  T 

IMPL 

6.0 

6.0 

3 . 3 

2.0 

8.0 

3 . 3 

6.0 

10 

6.7 

4.0 

6.0 

6.7 

6.0 

8.0 

3.3 

OPER  AVG 

|  5.6 

6.4 

5.3 

4 . 3 

6.0 

4.4 

4.7 

5.7 

6.1 

4.3 

t" 

• 

IT) 

4.4 

4.4 

5.6 

r- 

• 

Y  B 

UNTS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

Z  E 

CLSS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

AA  H 

RELS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

BB  A 

SYST 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

CC  V 

TRNS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

DD  L 

IMPL 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

OPER  AVG | 0 . 0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

o 

• 

o 

0.0 

0.0 

0.0 

o 

• 

o 

0.0 

0.0 

0.0 
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APPENDIX  F6 

LEVEL  ZERO  DATA  SUMMARY 

SUBJECT  6  5  YEARS  EXPERIENCE 

********************************************************************** 
QUESTION/ I  COGNITION  MEMORY  DIVT  PRODN  CONV  PRODN  EVALUATION 


PROD- 

-CONT  1 

F 

C 

D 

F 

c 

D 

F 

c 

D 

F 

C 

D 

F 

C 

D 

A 

V 

UNTS 

6 . 0 

8.0 

6.7 

4 . 0 

6.0 

6.7 

0 . 0 

0.0 

0.0 

4.0 

6.0 

6.7 

6.0 

6.0 

6.7 

B 

I 

CLSS 

10 

10 

3 . 3 

8 . 0 

6.0 

3 . 3 

0.0 

0.0 

0.0 

4.0 

6.0 

6.7 

6.0 

6.0 

3.3 

C 

s 

RELS 

6.0 

6.0 

3 . 3 

4 . 0 

6.0 

6.7 

0.0 

0.0 

0.0 

4.0 

4.0 

3.3 

4.0 

6.0 

3.3 

D 

u 

SYST 

10 

10 

3 . 3 

10 

6.0 

3 . 3 

0.0 

0.0 

0.0 

6.0 

8.0 

3.3 

6.0 

6.0 

3.3 

E 

A 

TRNS 

4.0 

10 

6.7 

6.0 

6.0 

3 . 3 

0.0 

0.0 

0.0 

6.0 

8.0 

6.7 

8.0 

8.0 

3.3 

F 

L 

IMPL 

6.0 

10 

6.7 

4.0 

6.0 

6.7 

0.0 

0.0 

0.0 

4.0 

8.0 

6.7 

6.0 

6.0 

3.3 

OPER  AVG | 

7.0 

9.0 

5.0 

6.0 

6.0 

5.0 

0.0 

0.0 

0.0 

4.7 

6.7 

5.6 

6.0 

6.3 

3.9 

G 

A 

UNTS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

H 

U 

CLSS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

I 

D 

RELS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

J 

I 

SYST 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

K 

T 

TRNS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

L 

Y 

IMPL 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

OPER  AVG | 0.0 

0.0 

o 

• 

o 

0.0 

0.0 

0.0 

0.0 

o 

• 

o 

0.0 

0.0 

0.0 

0.0 

0.0 

o 

• 

o 

0.0 

M 

S 

UNTS 

8.0 

8.0 

6.7 

10 

10 

6.7 

0.0 

0.0 

0.0 

8.0 

10 

3.3 

4.0 

6.0 

3.3 

N 

Y 

CLSS 

6.0 

8.0 

6.7 

2.0 

4.0 

3.3 

0.0 

0.0 

0.0 

4.0 

8.0 

6.7 

6.0 

10 

3.3 

0 

M 

RELS 

8.0 

10 

10 

6.0 

10 

6.7 

0.0 

0.0 

0.0 

2.0 

2.0 

3.3 

8.0 

10 

3 . 3 

P 

B 

SYST 

4.0 

8.0 

6.7 

6.0 

6.0 

6.7 

0.0 

0.0 

0.0 

2.0 

2.0 

3.3 

2.0 

6.0 

3.3 

Q 

0 

TRNS 

6.0 

10 

6.7 

6.0 

8.0 

6.7 

0.0 

0.0 

0.0 

2.0 

2.0 

3.3 

6.0 

8.0 

6.7 

R 

L 

IMPL 

4.0 

8.0 

10 

6.0 

8.0 

6.7 

0.0 

0.0 

0.0 

6.0 

6.0 

3.3 

4.0 

6.0 

3.3 

OPER  AVG | 6 . 0 

8.7 

7.8 

6.0 

7.7 

6.1 

o 

• 

o 

o 

• 

o 

0.0 

4.0 

5.0 

3.9 

m 

• 

o 

7.7 

3.9 

S 

S 

UNTS 

4.0 

6.0 

6.7 

8.0 

10 

6.7 

0.0 

0.0 

0.0 

4.0 

6.0 

3.3 

4.0 

8.0 

6.7 

T 

E 

CLSS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

U 

M 

RELS 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

V 

A 

SYST 

4.0 

8.0 

6.7 

10 

10 

3.3 

0.0 

0.0 

0.0 

2.0 

4.0 

3.3 

2.0 

2.0 

3.3 

W 

N 

TRAN 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

X 

T 

IMPL 

6.0 

8.0 

3.3 

6.0 

8.0 

6.7 

0.0 

0.0 

0.0 

6.0 

8.0 

3.3 

8.0 

10 

3.3 

OPER  AVG | 4 . 7  7.3  5.6  8.0  9.3  5.6  0.0  0.0  0.0  4.0  6.0  3.3  4.7  6.7  4.4 


Y  B  UNTS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
Z  E  CLSS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
AA  H  RELS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
BB  A  SYST  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
CC  V  TRNS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
DD  L  IMPL  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

OPER  AVG |0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
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APPENDIX  F7 

LEVEL  ZERO  DATA  SUMMARY 

SUBJECT  7  10  YEARS  EXPERIENCE 

********************************************************************** 
QUESTION/  COGNITION  MEMORY  DIVT  PRODN  CONV  PRODN  EVALUATION 

PROD-CONT  FCDFCDFCDFCDFCD 


A  V  UNTS  10  10  6.7  4.0  6.0  10  4.0  4.0  3.3  6.0  8.0  10  10  8.0  10 

B  I  CLSS  10  10  10  4.0  4.0  10  2.0  4.0  6.7  4.0  6.0  10  4.0  6.0  10 

C  S  RELS  4.0  8.0  6.7  4.0  6.0  6.7  6.0  8.0  10  8.0  10  10  6.0  6.0  10 

D  U  SYST  6.0  10  10  6.0  8.0  10  8.0  10  10  6.0  8.0  10  6.0  6.0  10 

E  A  TRNS  6.0  8.0  10  8.0  10  10  2.0  2.0  10  6.0  8.0  10  6.0  10  10 

F  L  IMPL  4.0  6.0  10  6.0  6.0  10  6.0  6.0  6.7  8.0  6.0  10  4.0  6.0  10 


OPER  AVG | 6 . 7  8. 7  8. 9  5. 3  6. 7  9. 5  4. 7  5. 7  7. 8  6. 3  7. 7  10  6.0  7.0  10 


G  A  UNTS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
H  U  CLSS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
I  D  RELS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
J  I  SYST  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
K  T  TRNS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  C.O  0.0  0.0 
L  Y  IMPL  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 


OPER  AVG | 5 . 6  6.8  7.3  5.6  7.6  6.0  3.6  5.2  4.7  4.8  5.2  8.0  4.8  4.8  8.7 

S  S  UNTS  4.0  8.0  10  6. 04. 06. 7  4.0  6.0  10  6.0  8.0  6.7  4.0  6.0  10 
T  E  CLSS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
U  M  RELS  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 

V  A  SYST  2.0  10  10  6. 08. 06. 7  4. 06. 03. 3  8.0  10  10  6.0  6.0  10 
W  N  TRAN  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
X  T  IMPL  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0 
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APPENDIX  F8 
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SUBJECT  8  10  YEARS  EXPERIENCE 
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APPENDIX  F9 
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APPENDIX  F10 
LEVEL  ZERO  DATA  SUMMARY 

SUBJECT  10  19  YEARS  EXPERIENCE 

QUESTION/I  COGNITION  MEMORY  DIVT  PRODN  CONV  PRODN  EVALUATION 
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APPENDIX  G 

SAS  SOFTWARE  DATA  ANALYSIS  PROGRAM 


001  DATA  DSATREL; 

002  INPUT  SUB  EXP  CF  CC  CD  CN  MF  MC  MD  MN  DF  DC  DD  DN  NF  NC  ND  NN 
003  EF  EC  ED  EN  TN  DSI  @@; 

004  IF  EXP  =>  10  THEN  GROUP  =  1; 

005  ELSE  GROUP  =  2; 

006  LIST; 

007  CARDS; 

008  PROC  PRINT; 

009  TITLE1  ' DSAT  RELIABILITY  STUDY'; 

010  TITLE2  'RAW  DATA  FROM  SCALED  RESPONSES'; 

Oil  PROC  UNIVARIATE  NORMAL; 

012  TITLE2  'UNIVARIATE  STATISTICS  FOR  INDIVIDUAL  INFO  ELEMENTS'; 

013  OPTIONS  LS=72; 

014  PROC  SORT; 

015  BY  EXP; 

016  PROC  CORR; 

017  VAR  EXP  CD  MD  DF  DC  DD  DN  ND  ED  DSI; 

018  TITLE2  'CORRELATION  DATA  FOR  DIFFICULTY  DIMENSION  AND'; 

019  TITLE3  'DIVERGENT  PRODUCTION  OPERATOR'; 

020  PROC  ANOVA; 

021  CLASS  GROUP; 

022  MODEL  DSI  =  GROUP; 

023  TITLE2  'F-TEST  OF  MEAN  DSI  SCORE  BASED  ON  LEVEL  OF  EXPERTISE'; 


